30 Days of Python - Day 16 - Scrape & Automate behind Password Protected Apps with Selenium & Python

hey there welcome to day 16 in this one we're going to be doing a little bit more advanced usage of web browsing technologies now what we did in earlier days was web scraping that's a type of web browsing oh well we want to do now is actually be able to log in to an actual service and do the things that aren't natively supported by a REST API in other words we want to replace the keyboard and mouse with Python so we're gonna be using selenium with Python but before I even talk about it let's take a look at an example so first of all we run our module called instead a PI that of course opens up a web browser takes me instagram.com that web browser is chrome and all of this is driven through Python and selenium so it's gonna actually log me into this service and look at I don't have my hands on there it did it it typed it it submitted the button all of those things then it moves me over to a user in this case the rock and then their actual profile or one other post there and it's actually gonna go ahead and download some items onto my local system that's related to this post and of course something else that we cover is not just the item downloading although that's really cool I don't actually need to be logged in to download those things but another thing that we cover is automated liking of the post and automated commenting which is really the reason that we're part of the reason that we're here but if we see this I got all these images they were all just downloaded as well as the videos themselves as you can see right there now we need to get selenium installed so if you're on a Windows jump to this timestamp right here takes about five minutes to get your windows set up not really that complicated to do but once you finish that you'll come back and join the rest of us if you're on a linux go to this link right here and follow their installation guides for the Linux distribution that you have if you're on Mac we're going to go ahead and install it right now the first thing I'm going to do is actually go to my terminal and just install it globally now the reason I'm installing it globally is to just make sure that I actually have selenium working and working correctly so I'm gonna go in and jump in to Python 3 - em hip install selenium no big deal there of course I will use this in a virtual environment soon but I wanted to install it globally just to verify that it's actually working so jumping into Python 3 and of course I'm using Python at three point two point eight but this same set of process works for I think every version of Python 3 and then it most likely will work in Python 4 as well so inside of Python 3 I'm gonna go ahead and import I'll go do from selenium import web driver so the web driver module class has a bunch of different types of drivers so if we do web driver dot Chrome this is the one we'll end up using but there's one for Firefox I think there's one for Safari just tech the selenium Doc's for more if you want that and then we're gonna go ahead and hit enter now there's a chance that yours will jump open like this it'll actually open a new web browser for you but more than likely you're not gonna have what's called the chrome driver installed you'll see something like this so this error here is to be expected I didn't actually install a chrome driver all I have on my system right now is Chrome for me to use it but not the individual chrome driver for me to do programming with so let's go ahead and install that now let's exit out of the Python shell clear this out and I want to install it using homebrew so brude SH is where you're gonna get install it and you're gonna go ahead and just copy this link right here now I already have it installed and you might as well so if you do just go ahead and run brew update just to make sure that your system is up to date now we're gonna go ahead and do a couple things related to chromedriver you might be tempted to run brew install from driver but unfortunately that's not how we install it from this version of homebrew or specifically for Chrome driver so I have to do brew tap homebrew tasks and then once I let that finish I have to run brew cask install Chrome driver and you're gonna use brew cask uninstall to an installed chrome driver as well we hit enter now this is gonna be a system-wide installation so once you have it installed you don't have to reinstall this unlike let's say for instance selenium where you might have it in different virtual environments chrome driver is just something that selenium uses it's not something that they like sort of work hand-in-hand together with we can clear that out and now what I'm going to do is try it out so we'll go ahead and into Python 3 I'll do from selenium I'm going to import the web driver and then web driver dot Chrome it enter and what this does is actually open up a web browser for me well that's pretty cool so I can also do something like this where I say browser equals to that web browser and it opens up another instance of that web browser might so notice that there's two now I can close out the one that I started with and now I can go ahead and do browser dot get HTTP slash slash CF e dot Sh and before I hit enter I'm just gonna move over a little bit so I can see the web browser and I hit enter and what do you know it actually changed that web browser for me and it's also telling me that Chrome is being controlled by automated test software so that's pretty cool so that's actually how I can go ahead and open up a web browser and run with it now for a little while will actually have the web browser open but what we can do is run with something called headless web browser where it doesn't actually open up a web browser at all it just runs as if it's going to open up a web browser but we just don't see the browser itself which is what you'll end up doing with automated tasks if you just did the Mac installation and you want to skip windows just go to this timestamp right here so let's go ahead and jump into PowerShell first and foremost I have Python three point eight point two installed so older versions of Python have actually worked with this method and I would imagine newer versions will as well let's go ahead and install selenium so python - module pip install selenium and while that's running I just want to point out I'm actually using the Chrome web browser here and that's what I'm going to be using with selenium there are other web browsers that it supports that definitely check the docs if you're interested in the other ones like Firefox or edge so I'm gonna go ahead and jump in to Python shell and then I'm gonna go ahead and import the web driver module so from selenium import webdriver and if I just do webdriver dot Chrome and hit enter I get this error right the chrome driver executable needs to be in this path so you can actually copy the link that comes here it's actually really handy all you need to do is and then right-click on it and then you can go to the Chrome web driver here and then go to all versions available and downloads and then you're gonna want to get the version of chrome that you have right so like I just updated to version 81 so if you're not familiar with what that is you can just click on the right-hand side with these three little dots go to help and then click the about I know it went off the screen there for a moment but there's my version of Chrome right there if you don't have the correct version of Chrome installed you're gonna see this error when you try to go ahead and use chromium so let's go ahead and download this chromium is just the driver that's gonna allow us to use Chrome with selenium so I'm gonna go ahead and download this driver here it's Windows I just whatever version of Windows is available that's the one I'm gonna use so in this case it's just a 32-bit one that's okay I've tested it it works so with this one we're going to go ahead and show this in our folder and I'm going to unzip this by using 7-zip and I'm gonna use the command for extract files I should had to use this command and change the location of it specifically for permissions related things so I'm going to go ahead and just change the location to my C Drive into the local disk so in here and I'm going to put it into chrome driver and I'll say ok and let's just verify that that's there by going into the C Drive and there's chrome driver and there it is that's the actual application that will run so now I need to add this into my path in my environment variables so go ahead and hit the Start menu there and go into and edit the system environment variables of course if you hit the Start menu and start typing it will do a system search for you hopefully you know that we're gonna go ahead and click on environment variables here and this is what should come up I'm gonna use this in my system variables so I'm going to go ahead and find my path system variable I'll go ahead and hit edit and I'm going to create a new one in this case it's just adding in chrome driver right so the same name that I used here is gonna is going to be the name that I'm gonna want to use inside of my environment variables there and I'm gonna go ahead and move this all the way up to the top probably not a hundred percent necessary to do that but I like whenever I add a new driver in there I just put it at the very top okay so we save that say okay to everything I'm gonna go actually close out of PowerShell just to refresh my session and then I'm gonna jump into Python again we'll go ahead and do from selenium import web driver web driver Chrome so the Chrome class we hit enter and a web browser actually pops open for me notice it's not actually anywhere but this is actually giving me a real web browser that I can I can actually work with we can just test and verify real quick here I'm gonna go ahead and run that again but I'm at this time when we go ahead and call it browser equals two and then I'm gonna set it equal to Chrome's webdriver dot Chrome and then that's going to open up a new browser and then like magic I'm just gonna go ahead and type out browser get and we'll do HTTP see if he Sh hit enter and what do you know that's pretty cool we now have a way to browse the internet using Python code there's a lot more we need to do for this so let's go ahead and get started all right so let's go ahead and open up vs code make a day sixteen directory if you haven't already and inside of here we're gonna go ahead and make our first file I'm gonna call this Google pi as in we're gonna be doing a Google search to start out and then I'm also going to create a virtual environment so let's go ahead and initialize a virtual environment open up terminal and we're gonna go ahead and CD into day 16 and of course we'll do Pippy and the install selenium and I'm going to be using Python 3.8 okay so while that's running I'm gonna go into my Google DUP I file and we'll just go ahead and say from selenium I'm gonna import the web driver okay so with our virtual environment done I'll go ahead and activate pip EMV shell make sure I'm in there let's go ahead and close down that Explorer so instead of Google dot hi I'm gonna go ahead and declare my web browser so of course it's webdriver Chrome or Like dot Firefox if you were using that one so this I want to actually navigate to a URL so the first URL I'm going to use is let's go ahead and jump into google.com it's really simple it's just browser doc get that URL so if I run this it's gonna open up the chrome driver version of Chrome and navigate to this URL let's go ahead and try that out so Python - I googled op I I hit enter and what do you know it navigates there now the reason I actually use the interactive shell has to do with the fact that now I can actually use the browser itself so I can say browser dot git HTTP duck duck go calm hit enter and it actually navigates to DuckDuckGo it's going to advocate wherever I want it to really cool okay so the goal here is actually really simple we just want to run a Google search that's all we're gonna be doing with this portion now the reason for that is because that's the fundamental underlying thing of everything we can do with selenium beyond just the web scraping portion so of course it can get complicated because of how these websites are designed but before we get to the complicated stuff let's start with the easy stuff now the first thing I need to do is actually enter my search query whatever that's going to be right so to enter a search query we have to look for an input with the type of text so this actually renders out that text book so once that test box is there on an HTML page that's the element that it is this is the HTML element itself you might be familiar with this hopefully you are at least a little bit but what this does is it renders out this now each HTML element every single one whether it's an input type of text or a text area most of time these things have what's called attributes so attributes can be stuff like a class or a CSS class an ID or a name so in our case I actually want to find out what this name is typically when you see these kinds of inputs whether they're just a raw input or a text area you will actually see a name that's associated to it that name makes it really easy to identify that's the element that we want the reason it's easy to identify it has to do with number one how selenium works number two those names are going to be a lot more reliable than their class names or even their IDs right because it's really easy to change a class name the actual HTML element name is not as easy because that often relies on the back end so the back end actually has to do something with whatever this name value is where the class and the ID are often more corresponding to how it looks versus to how it functions name is often how it functions so that's a cool little thing to know if you're not familiar with it so what we need to do is actually find out what the name is of this Google HTML element now the easiest way to do this is to actually open up Chrome whether it's from selenium or just a regular Chrome web browser right click or control-click and inspect the element you want to see now every once in a while I'm going to zoom in on the elements console here every once in a while when you do that inspect element it doesn't bring you to the correct one so you have to you might have to do it a couple times just to make sure you're at the actual right element and it's also easy to denote because when I scroll over these different elements it actually highlights what I'm looking for but yet again I'm looking for the input class itself or the input element itself and and this is it right here so I'm gonna go ahead and edit this as HTML with ctrl click and I'll just copy this entire thing and I put it under my notes here okay so as we see here we've got the input class this was what I discussed before up here we've got an attribute of max length and hey look at that the third one is the name of Q so we can we can be pretty confident that all we really need for this element is this so I'm actually gonna run with that I'm going to assume that that is the correct thing now once you see something that you know and you're familiar with that's when you test it you don't try and look for the best possible option right off the bat instead you should test as soon as you can so you can get things going as fast as you can so now the question is I have my input element let's go ahead and call this the search L how do I actually grab that right so I know that I want the name equaling 2q how do I actually grab this search element well the documentation for selenium definitely shows you how to do this so in the webdriver API we can scroll down and look for remote webdriver and then if you scroll down from there you'll see a bunch of different methods you can run on our web driver element or the instance of the web driver which the instance of the webdriver is browser in this case so what I'm showing you is methods you can run on that browser class now we can also write out dir browser and that will also show me all of those same things right so that's pretty cool so what we actually want to do is use something called find element by name this is in both places of course this is in the documentation it shows you a number of things but we've got find element by name and again the reason I know that that's what I want is because we want one of the elements that has the name of Q so I'm going to go ahead and just say browser dot find element by name and we can use Q now if you're actually familiar with CSS you can do another one let's find element by CSS selector and you can use a CSS selector say for instance like h1 that would be whatever the h1 element or multiple elements by just typing and s there either way both of those things will work and it's pretty cool if it doesn't find it it will run an exception which we'll see a little bit later once we get this down okay so now that I've got this let's go ahead and exit out of my interactive window here or the interactive Python shell and I'm gonna go ahead and print out what this search element is let's run the interactive shell again and this time it's gonna open a brand new window for me I do still have this other one open so I'm gonna close that one out I don't need it any longer but now it actually shows me this element here right so if that element was not found it would run an exception saying that this element is not found kind of makes sense okay so what is it that I'm trying to do here I'm trying to send text to this element you do this by using something called send keys so if you do search l dot send keys and then whatever text that we want to use so I'll go ahead and say selenium Python okay so this is well send I'll just do it on my interactive shelf right now and hit enter let's look at the browser that I've got open whoa it's already there okay so I'm gonna just put this towards the bottom and I'm gonna bring up my interactive shell a little bit so we can see a search box as well as interactive shell let's try that again what do you know it's gonna keep putting it in there that's pretty cool all right but I actually don't want that I just want the one-time search so I'll go ahead and close that out and I'll leave in the selenium Python okay so now what we need to do is actually find the the way to actually submit this form okay so submitting forms is going to be one of two two types of elements the first one will be input type of submit everything else doesn't really matter every other attribute doesn't really matter another one you might see is button type of submit so those are a couple you can also look for potentially a form element I'm not going to go in this one right here but you can also look for a form element and submit the form directly but these these right here are going to be buttons or have the ability to do what's called a click event so I can actually click these things so let's exit out of the interactive shell I'm gonna run it again this time it should navigate to Google and then actually put in the input now for some reason it didn't put the input in and it runs some sort of error that has to do with timing right so it's not going to be able to find any element if it goes too fast so you might actually have to use a way to just slow this down a little bit this is like the fastest most easy method to do it and we just do time dot sleep and we say the number of seconds we wanted to sleep before it actually runs the next part this will become a lot more important when we do more advanced stuff but now we need to find that that button that element for that button so of course in this case it's just the Google search button so I go ahead and inspect that element there and yet again I've got the wrong element selected if I defaults so I'm gonna inspect it again and what you know here we go there's my input class itself I added this as HTML I copy this whole thing paste it in here I've got a class that's not what I want I've got a value that's not what I want I've got the Aria label that's not what I want hey what do you know I've got a name that's interesting but that's still not what I want to actually want the submit type because that name could change it's certainly possible in this case the name could change the reason for that is because the type for submit button will override whatever the name is the name doesn't matter nearly as much with the submit button as it does for the inputs that are actually taking in text because the submit button just needs the event of somebody clicking on it so let's go ahead and get rid of all of these other things we've now found our input type of submit okay so now how do I actually find this submit button well it's also fairly simple we're going to be using the element by CSS collector or a selector rather and I'm going to call this the submit B tn element and i'm going to set this equal to my browser dot find element by CSS selector and if you're familiar with these CSS selectors you'll know that there's one for input so you can use an input like this and we can say type equals to submit and that should actually find us our submit button now we could also use the same thing for name equals to Q all right so that actually solves this up here if we were to want to do it that way as well but I'm gonna leave it in as type being submit and I'm also gonna go ahead and print out submit dot get attributes let's say for instance name I just want to show you that yeah you can get the other attributes on here so doing web scraping is certainly still possible with selenium in fact it's probably more robust doing web scraping than anything else because we can know that all of the javascript is loaded before we do anything and we get all these other session related things okay so I'm going to go ahead and run that without anything on the submit button element I just want to make sure it's there it's the correct one we're getting some attribute name and it will take a few seconds because I did time dots sleep and now I get button K and of course that's what a what it was when I actually grabbed that element but that's nice to know that I've now found it so what I can do is run the click event on here on this button itself so submit button click okay so again I will add another sleep method here so we can see all of this happen so I'll do sleep up to and we're going to exit out of the interactive shell again I'm going to close out some of these sessions that I had open from from you might have all these open as well and I just want my one krump session open okay so now that I've got this let's go ahead and run it this should open the web browser for me it does it will wait a second before it's entered text it does and then it waits a couple seconds before it submits it and it does look at that how cool is that I think that's pretty sweet okay so we now have some of the fundamentals down for this now what we just did inside of this Google one is not any different than doing password entry right so all we did was text just a query but we did it based off of the name so in theory we could use this exact same method to open up a place where we need to login to and I keep that session open as well now we're going to go ahead and login to a service now whenever it comes to using passwords or keys you want to be as secretive about them as possible so the very first thing I'm going to show you is a pretty awesome built-in way that Python has to make it as secure as possible so inside of a day 16 I'm gonna make a new file and I'm going to call this instead of PI because we will be eventually logging in to Instagram but I wanted to introduce you to the git pass library so you're gonna want to import get pass and I'll say my password he goes to get pass got get ass and I'll say what is your password okay so let's see what this looks like I'm gonna toggle the terminal with control tilde and I'm gonna go ahead and run Python insta PI right off the bat it asks you what's your password and I can say ABC it doesn't seem like anything happens so let's go ahead and just add a new line here and then I'll go ahead and print out my password okay so I'm gonna go and run that again and I'll say ABC I did type it I hit enter and it actually responds with the actual password itself okay so this is actually showing me how I recommend that you store your password going forward now the reason I'm gonna be using a comp I file has to do with the fact that I don't want to have to re-enter it every single time I'm showing you something I just want it to be there and usable now do keep in mind that if you end up using a comped up high file you're probably going to want to get in the habit of changing whatever that password is after you run anything that we're related to doing because I certainly do not want you losing your passwords because of doing something silly like leaving it in a comfy I file like I have right here so let's go ahead and get started actually doing this whole process I'm gonna leave this in here as reference for you but I'm certainly not going to be using that method right now so let's go ahead and do from conf we're gonna import the inste username and the insta password and then I'll go ahead and do from selenium I'm gonna go ahead and import the web driver just like we did with Google and I'll go ahead and say browser equals to webdriver Chrome and then I want to go ahead and say the URL is gonna be HTTP colon slash slash WW instagram.com and we'll just do browser get this URL okay I'm gonna close down this Explorer a little bit so I have more room to work and now I'm gonna go ahead and use the interactive shell to get in here mainly because I want access to selenium and a selenium session while I go forward so Python - I insta PI and actually instagram.com opens up so much like in Google when we did that one we had to look for the input with some sort of name alright so we did it by find element by name so I'm gonna go ahead and do that I'm also gonna make sure that time is imported so I'll import time and then do time dot sleep I'll give it two seconds and then I want to find the elements by name so in this case is gonna be the username element and it's gonna be fine element by name I'm gonna just take a stab in the dark and say username and then I'll go ahead and do the same thing with password now the reason I'm taking a stab in the dark has to do with how so many services name their login elements so let's actually take a look at what we've got on Instagram and I'll go ahead and inspect the element here and I might have to do this a couple times till you get the right one and then if i zoom in the console I'll probably see exactly what I was looking for and we've got an input here and what do you know the name is username and then the password let's go ahead and check that out password name is password as I suspected and then we need to log in button so go ahead and do the inspect element here so the actual submit button is what I'm looking for here and it's button type clot or the button class with the type of submit so this one's actually pretty easy as well if you don't remember it's simple we would call this the submit BTN element equals to browser dot find element by CSS selector this is one way to do it and then it's going to be the button and the type equaling to submit okay so I'm going to close down this Explorer here now that I've got these things I can just go ahead and run them on my interactive shell here okay so I've got those elements and of course we remember we have to do something like the element and then send keys and then in my case I need to send my instant username and then I need to send my instant password so password and sendkeys s word okay and then the submit button so simple we just call et enter click okay so again I'll go ahead and send my username with those keys and this is one of the reasons you should love the interactive shell it's in the password with those keys and if we look in our session username and password we're filled out and then I can I'm gonna go ahead and do imp time to sleep and I'll do three seconds oh I didn't import time so import time time to sleep three seconds and then I want to submit that and there goes okay so I don't necessarily need to have a sleep call here but I will do it time to sleep and this time I'll just do one point five seconds just to make sure my username and password entered and entered in a somewhat reasonable manner but what do you know there you go we just logged in to a service now that is super simple right like we just find the elements username and password and now we're inside of whatever element we want to use right so this is where we can do all sorts of things now before I get into the Instagram specific things let's just go ahead and get the HTML text and I'll just do browser and we want to get or find element by CSS selector for example and we can just use body and then this is going to be actually the body element would be body element and then the HTML text would be body element that get attribute inner HTML and it's lowercase like that with the capital HTML otherwise you will find that you have an error there and then I'll go ahead and print this out and I'll copy and paste this and there you go you've got all of the HTML element that you would need to parse so as far as web scraping goes going back to what we've done before which was requests HTML we can now use that to parse that HTML content right here which is another reason why I actually showed you requests HTML in the way that I did if you did that with us but it was essentially you just feed in HTML to request that HTML and then you can actually grab the HTML content in there you can grab stuff like the links the absolute links and so on so that's certainly something that I am interested in doing just not quite yet I don't think what we're at yet is that great so let's do the next part of this which is actually clicking the follow button for us like to do it automatically if all we wanted to do was web scraping behind this login all we had to do was grab the HTML document any HTML document and then parse it with something like Python requests HTML but what we want to do is actually add automated interaction with the service in this case it's Instagram but for example I want to actually click this follow button or I want to run a search these things start to get a lot more complicated because of how any given web application is designed right so we have the very basic and very common way of logging in right so this is unbelievably common across services but the follow button is not necessarily common right a lot of places use follow some places do not right so like your bank doesn't have a follow button most likely and so if we want to actually click this follow button that's where what we're about to do is really important but again if we just wanted to parse the HTML this is how you grab the HTML for any sort of CSS selector and then you've got that HTML there and you can go ahead and do the scraping that you want we will give some more examples of that shortly so let's go ahead and actually grab the L that I want to interact with so in this case it's that follow button so if I inspect the elements here of course you want to make sure that you're using a user that you don't already follow otherwise you'll see it looking a little bit more like this where you can message them so you want to make sure that that follow button is there otherwise this is not going to work for you and do keep in mind that Instagram does throtle accounts so if you start following ton of people it's definitely going to throttle your ability to do so I so anyways I'm in here now I've got this button highlighted I'm gonna go ahead and edit it as HTML I'll copy this element here and we'll bring it into vs code and paste it in as this documentation okay so what I see here is nothing that I think identifies this super uniquely in the attributes right these classes don't make a whole lot of sense to me maybe the developers that created this it makes a lot of sense to them and they are unique to this particular element but I'm not gonna trust that what I would rather do is just say button and follow like I want those two elements themselves now if you remember back up here I can use the CSS selector to grab any or all buttons and if I did find elements by CSS selector this would give me all elements that are a button and I can literally click all of those until I actually click the follow button I can also iterate through there to find what the text says to say if it's equal to that follow button that is one way to do it but instead I wanted to show you a little bit more advanced and that's called using XPath so XPath is similar to a CSS selector in the sense that it can grab things but I can also add some additional attributes to it that CSS selectors don't do as far as I know so what I mean by that is I can select something let's go ahead and just say my button XPath equals - / / that's a common thing and then button okay and then to make an identical call to the CSS selector we just pass in instead of by CSS selector do it by xpath and we pass in this argument here all right so it's literally that string so that's not that big of a deal here okay so it's gonna find the same ones as this but if I want to make it more advanced like I want my follow bTW and XPath in this case I can actually add in some brackets here these square brackets and type out contains parentheses inside it contains I use text parentheses and follow I'm going to go ahead and use actual single quotes here okay so now this will give me all of the buttons that contain the word follow now I do actually want to have one more aspect to this and put other brackets here and I'll say not contains the text all OH our single quotes again follow okay so this is two strings here or two conditions if you will that one saying that hey it needs to include the text follow the next one is saying that it does not include the text following that is certainly a button on Instagram through what I've done so we don't want to actually click on the following button but rather just follow so now I've got my follow button here and I should be able to grab that and this is gonna be follow BT ins or follow button elements will actually write out elements and again the reason its elements is because I have this s here if I took that s away it would just look for one single instance of that follow button which is not what we want so what I can do then is I can iterate through all of these so for BTN in follow button elements I can do time dots sleep this is critical right I don't want to overload the system this this is kind of like self throttling so we don't just jam their system up and get flagged that's what we definitely don't want to have happen which in my case my account is already it's like throttled itself so you know my following may may or may not work so we do BTN dot click and then we say accept pass so this is going to attempt it on all of those follow buttons and it's really just that simple okay so that's actually a method to grab these fall buttons we'll test this in just a second but obviously what I want to do is well I don't need to use this example by all means go ahead and try it out on your own but what I actually want to do is change it away from button and just having any element that contains this follow and not following and then I'm just going to click that element so in some cases maybe it is a anchor tag or an a tag so I want to have both of those things actually work so I'm just going to go ahead and do all elements with that star there and it's going to contain everything so I'll just go ahead and leave those as reference as well just for things that you could try out if you need so a tag button and then finally all okay so I'm gonna define this as click to follow or follow button you know whatever you want to call this function I'm going to pass this forward here and what I want to pass in is literally just the browser or the client session itself so browser and then that way I can I can actually trigger the current session whatever it is to use this click to follow method itself so by all means you can now use this exact same one on your Instagram it might also work on other services too so with that in mind let's actually go ahead and open up a session here so we'll do Python - I instead PI and of course it's gonna open up a browser log us in I'm not touching anything I don't want to touch anything I just want it to run if you move things around or click an element it might actually interrupt your service a little bit okay so I'm on Instagram now what I want to do is actually navigate to that new user right so I'll say new user equals - and of course in my case I'm just going to grab that URL that I had which is just simply instagram.com slash Ted and of course if you've never seen TED talks I highly recommend it next I'm going to go ahead and do browser get and let's call this the new user URL okay so I'll just go ahead and copy and paste these into my interactive shell and of course if I break down vs code I should see this actually happen in real time okay so here's my session going and BS code is right here so go ahead and run that and it brought me there no big surprise and then the next thing I want to do is actually trigger my follow button so I will just copy it first before I actually run it I'll copy this paste it here before I press Enter Oh collapse this down so we can see it in action I hit enter and all of them and sure enough it clicked it cool so this time it also grabbed followers it actually clicked on a button for followers and notice that I have following in there so I might need to add or append to my element here or my XPath lookup by adding one more one of these blocks here and take off followers all right so again we used every element here so I don't want followers I just want follow okay so now that I've got that I'm gonna go ahead and copy that same function and bring it into my current session so let's recopy this function copy and paste and then we're gonna run it one more time now typically speaking I'm probably not going to do it this way typically speaking I would probably just follow a certain list of users and then go off of that right but what we should see is I'm actually gonna bring this button down here in the session I should be able to actually follow all of those users down there as well not that I particularly want to follow them but I just want to illustrate the purpose here or the point of having something like this this is automating that interaction so yet again it actually called up the followers themselves which of course is not what I wanted so the shaming of these might actually not work as I anticipated so let's get rid of that for a moment and I'll just leave it back in as button just for a second this is what those trial and error things do from time to time okay so let's go ahead and try it just with that button again I want to make sure that this is opened and like to follow again and there we go so now it's actually going off of just the button so through your testing that is something that's very likely to happen which is part of the reason why I'm like AI want to show you how to do all or I want to show you how to do the button and it's worth testing those things out and in pit apparently with XPath you can't actually chain too many of these things together it's probably a way to add another condition in here but I don't actually need to use XPath that often because more than likely I'm actually not continuously trying to follow all sorts of accounts so notice that it stopped well the reason for that is because of how these buttons were dynamically loaded so if I run it again it's gonna now follow all of these new accounts and it does it in a speed that makes absolute sense for this particular user or like a real human right but again I'm totally automating this I'm not doing anything on here so if I actually go into my Instagram account on a different browser session I can see that my following account or who I'm following should be going up and it also depends on assuming them I click to follow is still running so let's go ahead and run that again and now it's gonna increment by one every second or so and I already have some followers which is hilarious because I actually this is just a blank account so people are following me back even though I you know don't have a real account so anyways that's actually how they create the follow button itself and actually clicking on that follow button and doing it in an automated fashion now if you were to get a little bit more advanced or want to get a little bit more advanced a couple challenges I have for you are to try and see all of the suggested ones or to keep running this until you follow pretty much everything it suggests you know how do you actually go about doing that I don't actually think it's that hard to do it's called a revert function I'll give you that hint but actually following all these these users is not actually what I want instead what I want to do now is actually go to any given user and scrape some of their data or actually interact with any given post to now we're gonna go ahead and use XPath to find individual posts for a user in this case the user I'm gonna be using is the rock so let's go ahead and grab that URL and this time I'll say the rock URL and obviously I want to navigate to there and let's make sure I do it after some time so time dots sleep.you seconds and so what I want to do is open up this rocks page and then find any given link in here so let's go ahead and start the process by starting the interactive shell of course saving step I okay so again I don't want to actually interact with that browser that comes up I clicked off of it if you interact with it you might run into some issues I just always keep that in mind okay so now we go into here and I want to inspect the element on any given item here so I'm gonna go ahead and go in inspect here and I have to do it again so I'm looking for the actual link itself okay and if i zoom in here we can see that the link or the a tag has an H ref and here's the link to that actual post alright so if you command click or control-click you can actually open up that post itself and that gives you what it is that we are looking for so that's actually the like format of the link and I know that all posts essentially start with the /p there and I know this only because I've used Instagram enough to actually recognize that pattern so literally if I use this post URL pattern I know that pretty much every single post as something like this alright like post slug ID and and obviously we're placing this with any given post so I know that this is the pattern I want to look for is instagram.com /p so what that means then is my host XPath string is going to be something like the a tang alright so we said it was an a tank we came back in here we see that as an a tank okay of course if I actually wanted to scrape all links I could I could totally use this and it probably would give me a vast majority of the posts but I want to get a little bit more specific so I actually want to use contains again they're using those brackets but in here contains so this time instead of using text we would use the @ sign HF and then /p / okay so what I know here let's close it out for the parentheses there so what I know here with any given pattern I know that for sure /p is in there I'm not positive if Instagram coms gonna be in there right as we could verify by even looking here Instagram comm is not in there a link itself for the href attribute only has the path to whatever that URL is so now we have a way to parse in theory all of the posts on any given page so that means then I can say post links equals to browser that find elements by XPath and that string and again it's gonna be elements not element okay so that's going to be all of those links so I can say for well let's go ahead and just grab the first iteration of it so I'll go ahead and say post link equals to be post links zero let's just be a little bit better as far as our parth on code is concerned and we'll just say if the length the length of the post links is greater than zero then we'll have at actual post length or a post link itself most link is not equal to none then we're gonna go to browser that get that post link hey real simple we're gonna go to the rocks page and then do all these things now I'm already at the rocks page so I'm already at line 57 here so I can actually just copy and paste these other lines let me go ahead and copy this paste it in there get an invalid syntax error here just make sure that these are correct so I'll just copy this and then we will one and we're getting none type for our post links okay so it looks like we have nothing as far as our a tanks are concerned so we are gonna have to probably out there yo ha ha there's the problem they should say post link you and we tried it again okay it looks like I should be able to navigate to that link so I hit enter and one more time invalid URL must be a string yes of course yes this right here is the element so this should be the post length element not the post link that was a silly mistake okay because again right here is gonna give me all of the elements this will give me a single element so all I need to do here to actually grab the post link or rather the post eight ref we grab that element and then we do get attribute eight ref and then we can actually go to that link right here okay so let's go ahead and try all of those again and that time it actually changed the browser I don't know if you saw it refresh but there there goes it actually changed the browser out and now I can actually work with whatever this link is so that's cool so the next part of this is actually grabbing the data that's in there right so again I'm still in that browser session so I can grab whatever the HTML data is in this case I just want the area where the image or you know where the content actually is not the comments with the content so I go ahead and inspect the element here and I'm just looking for the main area for this div class right and I'm just gonna keep going down and you know it's a little bit tricky to actually figure out which ones which what we actually need but what's not tricky is the element of video and image all right so video and image those are very standard so what I'm going to do instead is I'm going to copy these files themselves from the source that there they are okay so what that means then is I'll save videos or video how about we call it video elements or video else is browser find element we could still use XPath and I'll just do video and I'll use that same idea for images and this one's just IMG that's the image tag for any given image okay so now that I have these two things let's take a look like grab this paste this in here and video elements I have at least a single video element and image elements we go look at that looks like I might have some of those as well and I probably need to actually use SS here that way I get all of those elements and there we go giving me more than one this time which was what I actually inspected so image elements this is gonna give me a lot more now no surprise here the reason being has to do with all of these other images and any other image that's on the page right so it's actually going to literally get all of those images which of course makes this a little bit more tricky as to what we're trying to accomplish here and how we can find where that actual location is all right so so right now it's just like it's just not that clear so that's okay I mean it's really up to you on how you'll want to go about finding or parsing what data is important to you what we can do is say for IMG in image elements I can print out the IMG that get attribute and SRC so this is going to give me all of the URLs to those actual images assuming that they have them and we could save these okay so I'm actually not sure if we have requests installed we do not I could use selenium for this part but I'm gonna use Python requests so exit out of that session pipping the install requests you might also want to install request HTML but the idea here is we're gonna actually download it save these items here and I'll go ahead and put in let's start that session again who's gonna open up a new session I closed out the old one let that one run and inside of insta hi i'm import requests I'm also going to import this and add it into my session here and again I notice I have all these images here so if I scroll down to the elements what I want to do is I want to grab the base name so one other thing I need to import goes down this here I'm also going to import OS okay so I'll just make an image directory here so IMG derp equals 2 OS path that der name of well so let's actually call this baster or rather OS half that absolute path of that file and then image dur equal so as paths that join baster and images and OS that make doors of that image directory and we'll go ahead and say exist okay true okay so now I have an image directory in here so I can do then is say well I want to grab the file name with doing OS path that base name of the source file so the actual URL is equal to this here the base name will give me in theory the base name or one of the base names for that it might not work perfectly but that's okay and then I'll go ahead and say I want to grab the requests so R equals two requests get that URL actually I'm going to make it stay open so let's try this a little bit different and I'll say with quests got get URL and we'll say stream equals to true and we'll set it as R and then make sure you indent here and they'll do try our arrays or status and exception we'll just say continue which means we go to the next loop like it will break the current loop and they just skip to the next iteration and then what I want to do here then is say for chunk in our content I want to actually write to a file so file that right each chunk and this is because there's the ability to actually write out videos right so we can actually download videos um so before this of course I actually want to have a file itself so I did f dot right so I'll go ahead and say with open and I want to have my file name and my image directory so I'll go ahead and say file path equals 2 OS path join this is going to be my image Derr or video directory and then the file name opening up this file path and we're gonna do it as right bytes as F so WB is right bytes vs. read bytes or just W which would not write the bytes so our Barbie right bytes this will override what's also already in there as well so what this should do is actually save all of those images now I did a bunch of things so I'm not going to copy and paste this session instead what I'm gonna do is close it out and exit out of that session close at that terminal and let's try and run this again I'm gonna open up my Explorer here because what should happen is it should make a new directory for me and once it gets down here and then it should actually add in all of the images that it finds in that post so there's that and right so like right now a lot of these are not actually named correctly all right so they're not named correctly because of the question mark that comes after it but here's one that is a PNG file so we actually have an issue with these images let's go ahead and delete this move those to trash and it has to do with the file name that we created which was this right here so what I want to do is actually get the base file name and get rid of all of those other like parameters to them as far as the file name is concerned not the URL because we definitely want to still get the exact same URL but if we look at here's one example of a URL notice that the dot JPEG is right here and then it had all of these other stuff so I just want to grab and get rid of all of that other stuff so we're gonna go ahead and parse out these query parameters so this URL has a bunch of extra data on here probably to actually render out that image correctly now I want to parse it just for the file name I don't want to parse it for everything else so it's really simple we're just gonna use a built-in Python package so I'm going to go to the very top here and we're gonna import URL Lib dot parse I'm gonna import a URL parse or sorry not import but from your a Lib dot parse we're gonna import URL parse and so what I want is down into my image loop here I've got this file name but I want to go ahead and just say base URL equals to URL parse URL and then path so this should give me just the base URL path so I'm gonna go ahead and use that base URL path here and just grab the base name which we'll see in just a moment but I'll go ahead and print out what that base URL looks like and of course if there is no URL then we're probably gonna run into some issues here but let's go ahead and do this I'm not gonna start a new session instead I'm just gonna go ahead and do that import and then run that loop again which I should still have active right I should still be able to run this even though I just ran that session so getting no such directory or file I did delete them so maybe I need a Sri run the make directories here okay so let's rerun this loop you and hit enter okay so it's it's definitely creating things let's let's just take a look here in my images so now I should actually be able to render out these images but notice the filenames are much more accurate to a real jpg image and if I click on any of them actually get images here and it look like a lot of the users they don't actually look like the post itself and that's because the post I'm looking at is actually not related to a post but rather a video and so I'm going to do the same thing with videos so let's go ahead and copy this but this time instead of actually iterating through this way I'm going to turn it into a function called scrape and save and I'll go ahead and say elements and tab that in a little bit so this is going to be iterating through the elements let's close things out a little so we don't have to see it so this is going to be L and elements and get the attribute of SRC so going into our session let's make sure a video has that attribute okay so looking for that video we just need to kind of tab through here with all the various HTML content that we've got I found the video let's open that up a little bit and in the video of course does have an SRC so this this should work on images and videos and you can check out that base URL in there as well okay so scrape and save is our new function here I'll scrape and save I'm gonna do the video elements the base directory I still kept as the image directory that's no big deal I could call it let's call it data instead so dated err okay and I'm also going to say if OS path that exists then we're gonna go ahead and hit continue which would just take me to the next iteration in other words I don't want to resave the same stuff over and over again okay so now I should be able to go through the video elements and also the the image elements all right so let's close out our session again and exit out on our Python terminal as well run the interactive shell and let that go I probably should have deleted that image directory but it's gonna make a new directory so I'll delete the image one in just a moment but here's the automations going changes to that page it runs through the iterations it should actually find all the images find all the videos and store all of that for us there's a video actually loading it's gonna take some time because a video might be actually kind of big and depending on your internet connection if that that does absolutely make a pretty large difference so I'm going to go ahead and reveal this in the finder and there's the post and when you know there's the actual post with the video as well alright so it's the actual video and audio that's coming in there and it seems to be taking a good amount of time to actually go through what it needs to go through okay so it finished and mine says image Elle's it should be images Elle's I made that mistake no big deal the videos did download we can verify that the images will be a lot faster than the videos but as I iterate through these or just loop through them I can see hey check it out a thumbnail or actually that's the video itself there's another image there's another image there's there's a number of images in here right so so the videos the images they're all downloading on and that's how you actually go about scraping it so the key thing to take away here is this portion is how you actually download a pretty large content so this actually works for even really large files and you can change the chunk size or like how big it's going to be which can speed up your download process so this is a pretty common one I think that's actually the default size but you could check requests for that but this goes through each chunk of the file and then it'll save it to your local file of course if that's interrupted then you don't get the whole file but this this is a more efficient memory efficient way to actually download and save these files and of course actually getting rid of those URL parameters was really nice because then we can actually parse out that file name now in some cases there is no file name like it doesn't actually even tell you what the file name is right so in our case we were lucky that it had a JPEG or an mp4 in it but in other services they don't have either one so you're gonna have to do some more clever things to actually find out what either the file extension is or the file name is if it's images you can probably assume that it's a JPEG image those compress really well but if you need to you could use the Python image library or a pill to try and open the final results so after you save it you could try and open that in pill or pillow and if it opens then great it's an image if it doesn't open you can just ignore whatever that is so that's pretty cool and also using pill you can you can verify if it's a certain size so let's say for instance I want to ignore all of these smaller images right so in this case that's definitely users photo but the dimensions are 150 by 150 so we can use the python image library to verify verify let's type that right the size of any given image that's gonna be a nice little challenge for you it's not really that hard to do but it is something that I will challenge you to do on how you can actually do this in the request itself or shortly after the request so you could download it look it up and then delete it or even better then completely deleting it is just keeping a basically an empty version of that file in other words just writing the file but not actually storing any data to it depending on that actual size because then you can skip it in the future in other words if I actually ran this again it should not Reese crepe everything right because all of those files already exists so that's another key thing of why I did it that way is to make sure that hey I can download all these images and that's cool but I don't want to Reese crepe them over and over again one last thing that I would consider doing to make this a better scraper is to actually add the user who this sort of belongs to right so this file path adding maybe the user names here and creating all the directories necessary to make sure that I'm getting any given image for that particular user there's going to be a lot of things in there that aren't actually posts for that user but that's something you could you could definitely spend a lot more time on doing as well so that's pretty cool now that's of course just the scraping portion of this I do want to actually be able to like or maybe even comment on this as well that goes back to being selenium and getting some more advanced usage of how we go through with selenium so now that we know how to scrape and save any given image or video it's time to actually go ahead and interact with what's going on there now the reason for this is something that I think would be really cool to be able to do is use machine learning to let's say categorize the post or classify the post let's say classify because that's what you would call it classify the posts image or video and then comment in a relevant fashion long-term goal we're not going to do that in this series because that's just starts to get really a lot more complicated than I want to make 30 days of Python to be but that would be really cool and the reason I'm also showing you how to do all this stuff has to do with exactly that it's amazing how easy it is to create a service that can essentially download that data analyze it and then comment back on that all automated right so using machine learning would be able to classify maybe not every image super well but a lot of them and especially if it's like a specific person or a specific group of people that post very common things right like I know the rock comments or posts all the time in the gym working out you know you could classify that hey he's working out and then your automated comment could be some variation of like nice work out or hey way to go or make sure you buy my protein supplement or something like that something ridiculous like that and again this is this is for understanding that these systems are set up this way like they're not set up to prevent this from happening like which is which is interesting right so we are as even as beginners we are able to come in here and do a lot of these things all right so let's go ahead and start this session all over again now I actually am NOT going to run these scrape and save again or you click to follow right I'm just gonna go ahead and go back to the rock and take a look at that so let's go ahead and restart our session and exit out of here and go into insta so hopefully this takes me to the most recent post whatever that might be that's this portion right here that should give me the rocks most recent post as an individual detail and there it is okay so first things first let's go ahead and actually make a comment on inspecting the element for the comment button itself I need to open up the browser here a little bit sorry if this is going to be a little small I just want to make sure I can grab that specific element so I'll zoom in so we can see what this is so let's get rid of part of it like I want to zoom in too much on the browser otherwise we might lose it okay there we go so yet again we don't have anything related to the actual name of this textarea sticks area does not have a name but it does have a placeholder so let's go ahead and edit this element here I'm gonna copy this bring this into vs code and we'll scroll to the bottom and now let's take a look at this element right so Instagram does this on purpose they do not want you to be able to submit things automatically although it definitely still can be done okay so what I want to get is I want to grab the element so let's call this the comment l and it's gonna be browser dot find element by XPath this case is going to be one single element I believe there's only one element which we can test out in just a moment and I want to use the text area so let's go ahead and say my comment XPath string it starts with the two slashes and it's going to be a text area okay so the text area we want to use brackets and contains now again I'm gonna use an a tag or attributes this is going to be placeholder and this is gonna be ad a comment okay so I am NOT positive if this is gonna work let's go ahead and try this out and in theory I would grab that element and then I would send keys and say awesome something like that okay so let's go ahead and paste that in there make sure that that's correct let's just verify parentheses are closed at placeholder and the real okay so let's go ahead and find that element let's see if it's even there it appears that it is so let's go ahead and send these keys I'm not gonna press ENTER yet just because I want to see to make sure I'm gonna press ENTER now boom that's awesome okay like literally I can actually post it now so but what I need to do is actually find the next element which would be the button that's right next to it with submit but I'm actually gonna go I'm gonna go a little nuts here and I'm just gonna say submit BTN s equals two I'm gonna just try out grabbing all buttons that have the type equaling two submit or rather button with the type equaling to submit it and we can actually use the CSS selector for this we don't have to do it with the XPath let's just keep on that so now I'll go ahead and do submit if he ends equals two browser that find elements by CSS selector submit buttons okay so for PTN in submit buttons we're gonna go ahead and try ET and click accept pass okay so submit buttons let's go ahead and look at this oh this should be this shouldn't I rename this incorrectly or name this incorrectly this should be XPath that way nothing gets confused here and then this will be submit button elements or else go okay let's try that again let's just take a look make sure there's something in there looks like there is there's only one that's a good sign shocking that there's only one but there's only one so I'm going to go ahead and run it and I'm gonna copy and paste and in theory this will actually post for me I'll copy and paste it into a couple times and look at that loading look at that that is an automated comment for I go into this next part so let's go ahead and put the automated comments into a method itself called automate comment and we'll just pass in the contents here in this case I will say that's like something like that that is cool you know super generic of course if you want to get a little bit more advanced you could have a list of things and sending from that list it could be a really long list of actual content items you pass through there or of course using machine learning to initially classify with send the content and then make that comment and of course I want to add this into sand keys and then I want to make sure that I'm also separating these things out well enough so I'll do time dot sleep and then I'll do let's say like three seconds initially and then before I actually submit it I will also do let's say two seconds so let's go ahead and try this out now and I'll put this at the very bottom actually let's just leave it out and I'll run the interactive shell I'm just gonna go ahead and let it run and once it's done I'll come back cool so now at that post let's go ahead and try out this automate comment thing here and run that and [Music] I get element is not interactive interactable so I'm getting something with each element so with that actually going into the browser at all let's go ahead and try this out again it was working so it's possible that we need to oops there's a little problem error let's try that there we go that's what we we might have to adjust this slightly you okay and just to verify that this is definitely working and that time it worked okay so what I'm going to say is I will just go ahead and pass the browser through as well so that when I oughta mate this comment it should actually go through based off of that current browser session like I talked about before this time I'm gonna go ahead and delete that is so cool and let's try that again cool there it is okay so how we've got that let's go ahead and move on to automating likes so now what we're gonna do is just trigger I like event as in liking this post now this is cool and all but actually the main thing about this is actually getting the parent element so if we inspect the elements for the heart itself what we will find is that the heart is not a button but rather it's an SVG so you can't actually click on an SVG that doesn't do anything that won't actually run the event like it would if there's the button and again no real and identifying things for that button the thing that's identified is the Aria label alright so the Aria label of like that is an identifying and I also could go off of height because if we inspect the element on these other ones I get a different height although the same actual like itself or the same Aria label itself so let's go ahead and try this out into vs code now I'm already have my browser session going so make sure that you get yours going if you don't already so what I want to do is say the like heart SVG XPath equals two and this is going to be slash slash anything that contains and this is going to be at Aria dash label remember that at sign is for any attribute that's in there and then this case is going to be like and it's a capital L and we can close that out okay so I should get every single one of these right so let's call this all like arts elements and this is gonna be browser that find elements by XPath Essen in this XPath here okay so now I'm going to go ahead and iterate through these so I'll go ahead and say for heart element in all heart elements all like heart elements I'm gonna go ahead and print out the heart element I get attribute and hiked okay so we'll save that and I'm going to bring this into my interactive shell here you got invalid syntax oops I didn't mean to put a colon there but I did try that again and a lasat array through them a lot of them are 12 and then there's one that's 24 okay so we want the one that's 24 if let's call this H equals 2 that ages in height of course so if H equals equals to 24 then we want to get that parent button well that's going to be taking the actual heart element itself not the browser with a heart element itself or any element for that matter and then do dot find element by X path and then we just do dot dot so this takes me up one level that should be my parent button and I can try to click it and I can do except path or pass okay I also want to print out what that is just in case you now I'm gonna go ahead and run this for I press ENTER I'll collapse this down a little bit so I can see if that heart turns red we hit enter a couple times and doesn't look like we got what we were looking for I know why it has to do with this so we'll say if it's equal to 24 or H is equal to the string of 24 gotta love those strings okay yet again I'm going to copy and paste it and let's go ahead and just break this down again we go okay and hit enter a couple times Oh liked it so naturally there's a little bit of a flaw to this right here and that is what if they change the height just a little bit then perhaps we have broken this like button so what I'm going to assume then is that that heart will always be bigger than the other hearts so let's go back in a chrome in other words this one will always be bigger than these but as it stands right now if we inspect this element this is actually now an unlike label right so in order for me to test this theory I gotta unlike it make sure that it's unliked and now what I want to do is actually iterate through all the hearts find what the biggest number is and then iterate through them again and go off of that number instead of the one that I manually entered okay so in other words I'm just gonna run this twice and grab this here so we'll say max heart h equals to negative 1 because of course it's going to be bigger than negative 1 and i'll call this the current h so current h equals to the nth of whatever that h is and if current h is greater than max heart h then we'll set the max heart h or height equal to that one so now we would just come down here then and change it to being like that okay so in theory then this should actually give me 24 assuming that that like buttons actually even there okay so let's go ahead and run these again so yet again I made sure that I unlike this let's go ahead and open this up I'm just gonna run it and not look at it I printed out max heart H is 24 and there you go it's 24 well so that will definitely make sure that I'm liking that big heart button you can certainly like all the other comments but once you start doing stuff like that it really starts to feel very bot like every once a while maybe you like some of the other ones but the real purpose of this was to see how to actually grab the parent element and you can grab that parents parent right so like the grandparent element if you will by doing the same method so find element by XPath can also be used on other items right so you can use it on the CSS selector as well because any element that is selected using find elements whatever you can actually use find elements again as a sub-module basically to the other one okay so in general with this like I still want to you know create another function and say automate likes and pass in the browser and tab these over you might be wondering why did I actually grab these like elements twice well the reason I grabbed them twice is so I could iterate through them and in some cases when you iterate through it it actually goes quote-unquote stale so it won't allow you to reiterate through it so I just do it twice that's it I don't actually need the XPath twice though I can use the same one and then this one I can just say the large Hellmann's or this one being the large elements and of up to you but whatever you end up using just make sure you update that iteration and again they click event I'm just going to put it next to that and just do time to sleep and two seconds and again we want to be as human as possible cool hey there thanks for watching day 16 I realize that we cover a lot here and at times there's probably rather frustrating and that's because selenium is not as straightforward as many other Python packages because it's not just for Python and selenium itself doesn't account for what other users or other developers have done with their applications what we saw was well not so straightforward ways to actually interact with one specific web application and behind their login screen that of course is not going to work perfectly on another service even if they have a lot of those same features which of course that's a little frustrating but the idea here is that there's a lot of services out there that you can start to automate these processes with and something like Instagram it's a little nerve-racking that we can do something so effectively within a couple hours and we could do it across a number of users and on a regular basis it's kind of weird that that's even a possibility and of course there's other things that you might want to learn more about with selenium which is like uploading an image like actually clicking on something to actually upload an image to Instagram that starts to get a lot more complicated than what I wanted to cover here and that would turn this into a full-blown selenium course but more this is about more advanced usage of web scrape and using a web browser so even if there's only parts of it that you can really use now that's okay just realize that selenium is a very powerful tool that you can use in the future for all sorts of web browser type automations and testing so again thanks so much for watching hope to see you in the next one

Channel: CodingEntrepreneurs

Views: 19,691

Rating: 4.9553905 out of 5

Keywords: djangourlshortcfe2018, install django with pip, Python (Software), web application development, pip, django, beginners tutorial, trydjango2017, install python, python3.8, django3.0, python django, web frameworks, install python windows, windows python, mac python, install python mac, install python linux, pipenv, virtual environments, 30daysofpython, beginner python, python tutorial, selenium, web scraping, automations, instagram

Id: 3DCtaJvf6VA

Length: 95min 36sec (5736 seconds)

Published: Tue Apr 14 2020