Python Requests login and persistent sessions tutorial 🔥: the "Hacker" way | Python web scraping

a user of a the D 410 nickname asked me to make a video about authorization on websites using requests library and he suggests to use CV shoes.com as an example she with shoes as I understand is an issue that hides its prices until you're locked in this page has no prices I am logging in with my fake account and it redirects me to the catalog page with the price revealed ok the task is simple I have to login to the CV shoes.com website with the request library without using selenium webdriver to get product prices the task is simple but solution is quite tricky and because this website has a protection and I'll show you how to bypass it and now I want to create two Python scripts config py and main view I and here in the config py file I want to create variables to store my username and the password and the password is also the string something like this and now I want to import these variables into my main not py script from config import username password then I need an entry point of my script and now let's import the request library and look at the service responses if you need to install requests you can do it with the peep in store requests beautifulsoup and L XML in the main function I am creating a new variable the URL the URL I want to send the question it will be and now let's examine again the web page first of all I want to know on what requests my browser sends when I submit on the forum so I'm opening the inspector I need the network tab and I am submitting the form and there is login requests it's a post request over the 200 status maybe I was wrong yep post requests and server responded with 302 status code it means redirect the website redirects me to the catalog page and it uses sessions so I have to use sessions too so to perform all requests I have to use request sessions and I can do it with using the weave context manager I want to save open session to this session available and I am creating the response variable that equals to session variable and i'm kahlan the post method to perform a post request the first argument is a URL this one and the second argument is an oath parameter it's a topple the first element of this tuple is a username and the second is the password these variables and by default all requests library methods allow redirects so if the page will change its content I will know about that also I want to use the print function to print out the server's response response text the text property returns me HTML code of this page or I can just save the response text to HTML file and then just open it in Chrome maybe it's even better I am saving opened for writing index.html to the F variable I'm calling the right method and passing in to eat the response text I'm running the script can't import name username from config okay username again I've got index.html and I want and I want to use autumn's at a live server Adam live server will automatically refresh the page when the content of the index.html file will change just for convenience and we can see that nothing is changed I've got the login page again yep we can see it in the code - password input field and here a user input field and now I want to show the protection of this website and how to bypass it let's examine browser sequest again so I have parameters of the post requests to the login page post that was successful and the website redirect me to the catalog page so it's the response I need and first of all I need to check the body of the requests what data my browser send to the server and in chrome you can see a body of my request at the bottom oh so I need the form data section and we can see here my username the password key of this object is empty and there are mysterious challenge key and the response the challenge has no value and the response has a song this string okay it's a form data I think that it's time to examine the form to get a form again I have to log out log in page and this is the form and we can see here that the form has input tag with the user name ID and the name username in the key div and it has input tag for password and also it has two hidden input tags challenge input and response input and the challenge input has a video with some stain but as you can probably remember the challenge parameter of the quest body was empty and the response parameter had the value but now it's reversed the challenge has the value and the response doesn't and the data these two hidden texts were included to the request and now if i refresh the page 7 CC 9 is 6 the value of the challenge input has changed and it means that something happened when I submit the form so we can see here that the form is handled by du challenge response function it's a JavaScript function so we can just treat it ctrl F paste new challenge response function and this is the function ok what does it do I want to copy it and paste tomorrow Python script somewhere here so we can see that it has three parts and the first part is the concatenation of some strings we can see that it takes the value of the username input tag and add to eat accountant then it gets the value of the password input tag it gets its md5 hash and adds to this hash the column and then it adds to this string the value of the challenge input tag so the resulting strings looks like this username column md5 hash of the password column and the challenge value and then we can see that at the end of the function it gets the md5 hash from this entire string and this hash becomes the value of the response input tag password field and challenge field becomes empty and this form data this function sends to the server and now I want to reproduce this function with Python to do the same first of all do challenge-response function gets md5 hash twice for the password and for resultant string so I want to make a separate function to get md5 hash Python has ashleighb library so I want to import it and I'm defining a new function get md5 function that takes some string as an argument to get the md5 hash of the string I have to call the md5 function ashleighb md5 and it takes as an argument the bytes string so I am calling the bytes function and as an argument I am passing into it the S variable and I have to specify the encoding of this string it's a Unicode string so the encoding parameter is utf-8 and to get the hash I have to call the hex digest method and that will be the returning value configured md5 function let's test it the hash on the red word a da da for BD a ei for okay I have to get the md5 hash of the password so md5 pass variable will be the result of the md5 function password I'm adding the column then I need to add to a username the column is their name as we can see here and then I have to get the value of the challenge input tag and to get it I want to use beautifulsoup library so I have to import it from BES for import beautiful subclass I need an instance of the beautiful subclass it gets as an argument HTML code of the page so I need the response variable its text property and that'll be simple get requests without off parameter and I have to specify a parser HTML parser I want to use Alixe image and now I need to find the challenge input tag so let's go here again and the challenge input tag has ID challenge I am calling the find method I'm looking for input tag with the idea equals to challenge then I have to get its value let's test it local variable username reference before assignment it's weird okay I've got the challenge value and let's move on and at last I have to create the resultant string so the result will be a username email in my case plus md5 Pass variable plus challenge and then I have to create the response value I'm currently get in the five function and I am passing into it the result string or just copy it here and now I've got only the data and I want to send it to the server with a post request so data object it's a dictionary with the username with email as the value the password name password sorry it will be an empty string then challenge parameter is an empty string - and response is response and then I'm sending a request to the server it's all within the session session post URL data parameter equals to my data dictionary this one and here I want to use our post now I want to be print our post text use your name email username plus Collin Collin K username should be username because email has Collin at the end okay let's test it and I think it's done yep the prices we've got prices it's not login page and we can see it in the in the co2 for example it's done and the conclusion there is no security at the front end so if you like the video please thumb up and subscribe to the channel thanks for watching

Channel: Red Eyed Coder Club

Views: 44,934

Rating: 4.9101124 out of 5

Keywords: python, requests python, requests session python, requests login session python, requests session cookie, requests python login, python requests login, python requests session, python requests redirect, python requests cookies, python requests authentication, python requests post login, python requests session login, python request, python login to website, python requests, python login, python requests tutorial, python requests login to website, request python, login python

Id: wMf7LJn0k4U

Length: 22min 28sec (1348 seconds)

Published: Sun Jun 07 2020