SFTP in python with pysftp along with examples

hi friends welcome to 10 python series in this video we're going to talk about how to use the pi sftp python module to interact with an sftp server to perform actions like downloading files uploading files listing out the file names etc so since pi s ftp python module is used to interact with an sftp server i will set up a local sftp server using openssh here you can see in my services.msc there is an open ssh server running as a windows background service i have connected to my open ssh server using winscp and this is the root folder of this open access server actually i made my screenshots folder as the openstack server root and that's why you can see the same folder contents in my sftp server root this current sftp server setup uses private key authentication for securing the sftp connection if you don't know how to set up an sftp server in windows i have already created a blog post along with instructions on how to set up an sftp server in windows and how to set up password-based authentication and private key based authentication in sftp server in the blog post i also explained how to use win sap to connect to an sftp server so that you can browse files in the sftp server and perform actions like uploading and downloading i have also made a video on how to set up a windows based sftp server and how to use mini cp to connect to the sftp server so if you don't know how to set up the hftp server which i have created in this video you can go back and see that video and i will put the link of that video in the description all right first let's try to install the psftp python module for that i'm going to open my command prompt and i'm going to run the command python minus mp install pi sftp my sftp python module is installed in my python environment now so now we are ready to use pi s ftp python module so let's get started with coding and connect to sftp server using pi sftp i'm gonna take a blank folder and i'm gonna open this folder with vs code i'm gonna create a new file i'll just name it index.fine first i will import the pi sftp python module to connect to an sftp server using pi sftp the connection parameters required are the host name port username and the private key file path so let's try to create variables for that now sftp host equal to localhost sftp port equal to 22. generally sftp runs on port 22 username equal to the operating system user and the private key file path i need to specify the path of the private key file in this variable right in my previous video i've already created a private key file and i've saved the private key file in this username dot ssh folder as id underscore rsv and this private key is actually used for this sftp server so i'll just copy this private key file and paste this file in the same location of the script so i'll open this folder and paste the private key file here so now i have the privacy file in the same folder so i can just write dot slash id underscore rsa if you want to mention an absolute file part you can also do that so if i want to mention the absolute file path of the private key file i'll just copy this folder path and paste the folder path here so now this is the absolute folder path of the id underscore rsa file which is a private key and since i have black slashes i'll just write r so that i don't need to write the double backslash but for now to make things simple i just keep id underscore rsc because the file is present in the same folder as the python script since we have our connection variables declared let's try to connect to the sftp server using pi sftp to connect to the sftp server using pi 60p you need a sftp connection object so let's try to create that i'll just write with pi sftp dot connection and inside the connection you have to be the parameters so host equal to the sftp host and the port equal to the sftp port and username is equal to the sftp username and the private key file path is private underscore key equal to the private key file path and let's alias this connection object as sftp and let's start the with block so inside this with block you will get an established fire safety connection as an object called sftp so inside this width block you can do whatever with the sftp connection so since we are connected let's try to create a simple print statement called print connected to sftp server all right so let's try to run this python code and when i run this code i got an exception called no host key found this happens when you don't have the sftp known host file or your sftp host is not included in the known host file so to mitigate this problem we have to create connection objects so let's try to go ahead and do that i'll just write connection options equal to 50p dot connection options and here in the connection options and in this connection options set host keys none so using these two lines we are ignoring the known host file check while sftp connection using psftp it is not recommended to skip known host check as it reduces security but in our example we are just doing this to avoid the known host file check since i've created a connection option object let's try to use that while creating the connection object so here i'm going to give a new input called connection objects and i'm going to specify the new connection options object which i have created just now now since we have ignored the known host file check let's try to run our code again and here you can see connected to the sftp server so just with a single line of code you can use psftp to connect your sftp server so if you are using password-based authentication to connect to an sftp server instead of writing the private key input you can write password input and write password equal to the password which you are using while creating the xftp connection all right since you have finished connecting to the sftp server let's try to change the working directory of this ftp server so here in wind sap you can see this is my sftp server root folder and i have a subfolder in sftp server and this is called remote folder so i want to change the root folder to this remote folder folder path so i'm gonna use the cwd function of sftp connection so that i can change my working directory so here i'm gonna write sftp dot cwd and here i have to write the new folder path which is going to be the sftp root for this connection object so here i'm gonna write dot slash remote folder so i'll just save it and if you want to print the current working directory i can just write sftp dot pwd which is print working directory and here i'll just print this so that we can see what is the working directory so before changing the working directory let's try to print the present working directory so we can see the change of the working directory by using the cwd command so i'm printing the working directory before changing the work directory and printing the working directory again so let's try to save it and run this and here i got file not found error because i have misspent the folder name here you can see the folder name is rmfo ldr but i wrote dr so let's try to correct that i'll just change the folder name rem folder and then run this again i'll just clear the output and run again the first working directory was slash and the next working directory was remote folder since we have used the sftp dot change working directory to change the remote working folder all right now let's try to find out what are the list of file names in a remote folder using the list deer command so if you just use sftp dot list dip you'll get the file names in a remote working directory so let's not use that now i'm going to clear this code now i'm going to write sftp dot list d what listing does is that it will return the list of file names in the root folder so i'll just get into the variable called x and i'll just print this variable if you want to get the file contents inside a remote folder you can write x equal to sftp dot list deal and in the input you can give the folder path so if i write the folder path as the remote folder i will get the file links inside the remote folder as opposed to the file links in the root folder so i'll just print the x again so let's try to save it and run this again i got the error because the folder name was wrong so i'll just write remote folder this is the correct folder name i'll just save it and run this again and here you can see this is the files inside the root folder and this is the files inside the remote folders folder of the remote server so here you can see inside the remote folder i've got only two text files and you can see the two text files here and inside the root folder i've got some three files and a single folder and here you can see about the three file names and the single folder name so this way using the list command you can very easily get the list of file names and folder names inside a folder in a single line now let's talk about another function which is list div attributes which gives you the file names and metadata of the file names also so let's try to use that now instead of using list due i'm going to use listing attributes and i'm going to remove this below section so now using this listed attributes i'll get the list of file name metadatas and i'll just print it now i'll save it and run this and here you can see i got file name metadata objects and in each object you can see the attributes which is size the id of the user the id of the user group the mode of the file and the last access time which is in a unix epoch timestamp and the last modified time so this metadata of the file can be accessible using the list just attributes so instead of printing the objects i can also print each metadata item by using a simple list compression so i'll just write y for y in x and instead of printing y and this print a list of the items present in y y dot file name mode size and the modified time so if i just run this here you can see the file name mode the size invites and the last modified time in e-mix epochs type strap so now let's try to upload files into sftp server directory using the put put d and put r functions so all these functions are similar but now let's try to understand what is the difference between these functions so if you use the sftp.put function you can upload a single file into the remote server and if use the put d function you can upload all the contents of a directory in the remote server directory and if you use the foot or functions you can upload the folder contents into the remote folder and all the subfolder of the local folder will also be copied into the remote folder so let's try to use the sftp.put function so in order to use that function i'll just create a file so that i can upload into the remote folder i'll just create a file i'll just name it abcd.txt and i'll write some text in this so now i've got a file abcd.exe and i want to upload this to the remote server so i'll just write sftp.put and here first i have to mention the local part the local path is abcd.txt and you can even mention the absolute local path like c users and whatever and then you can mention the remote path if you don't mention the remote path it will be stored in the root folder and one more flag which i want to show you is the preserve m time equal to true so what preserve m time equal to true does is it will preserve the modification time of the file in the remote folder also so i'll just save it and run this and if i refresh my remote folder you can see what my abcd.txt upload it into the remote folder and if you want to upload abcd.txt into the another folder of the remote server i can mention the remote folder path also so if i want to upload this file to the remote folder i can just write remote path and here in the remote path i can write the full path of the file inside the remote server so let's write remote folder slash abcd.txt or else you can write some other name i'll just write xyz dot txt so i'll just save it and now my abcd.txt will be uploaded into the remote server but the path of the file will be remote folder slash exposure.txt so let's run this and if i open this remote folder here you can see xyz.txt is uploaded here so this way using the put function you can upload a single file into the remote folder all right let's try to use the put d function so put d function will upload the folder itself into the remote server so here for example i'll just create a new folder i'll write abc i'll create some three files in this so all right i've got a new folder called abcd and i've got three text files in this abc folder so now let me try to transport these contents inside the abcd folder into my remote server sftp folder so for that i have write sftp dot put d which uploads the folder contents into the remote folder so here i will do the local path which is abc you can even give the absolute folder path like c users abc so on like that here i am saying preserve the modification time of the file contents and i am doing the remote path equal to the folder path of the remote server so if i want to copy to the remote folder i'll just write the remote folder so now this file contents will be transported into this remote folder i'll just save it and run this now and here i don't know if it is particular to windows but i can't give relative path using put d functions so i'll have to give an absolute folder path so let's open this folder in windows explorer and copy this folder path and paste that folder path here so now i've got my folder path which is an absolute folder path and now i'll run this function and here you can see the files have been transferred to the win sap server so if i just open this remote folder inside the root folder and just refresh this here you can see the new text files being transported into this remote folder so using put d function i have uploaded the folder contents into the remote server if i don't mention the remote path the file contents will be transported into the root folder put our function is also similar to put d but if you have subfolders even those subfolders will be transported to the remote server now let's try to get to know how to download files from sftp server directory using get get d and get r get can be used to download a single sftp server file and get d can be used to download all the remote folder files in the local folder and get r is like recursive download where you can download all the remote files into the local folder and you'll get the sub directories also from the remote server so let's try to use sftp.get i'll just write dot get here i'll do the remote path first so i'll just write a remote file name and then i can write preserve m time equal to true so that i'll get the file modification time the same as the remote server so in order to write the remote folder path and let's see the file name here here i have a file called abcd.txt in the remote folder or else i'll just use this file name screenshot1.png so i'll write screenshot1.png so this is my remote file name and i have to download this file into my local folder since i have not given any folder path so let's run this here and here i got the screenshot1.png file from the remote folder you can even download a folder which is nested inside the remote folder so if i go to if i want to download a file called remote folder slash jhj i can just write dot slash remote folder slash jhj.txt and let's save it and run this now yeah the remote folder name was wrong remote folder was this file name so let's save it and run this again and here you can see i've got the jhj.txt so this way using the sftp.get function you can download a single remote file into your local folder but if you want to change the file name while downloading you can even specify the local folder path so i'll just write local path equal to fgh dot txt so i'll just save it and run this and here you can see i've got a new file called fgh.txt i can even save in some absolute folder path something like abc slash pl o dot txt so let's save it and run this and inside the abc folder i get a new file called plot.txt so you can even download the single file and place it at a different location also using this local path input all right now let's try to use the sftp dot get d where you can download the whole directory of the remote server and paste the contents into our local folder so let's create a new local folder i'll just name it uyt so this is my new local folder and i want to download the contents of this remote folder into this local folder called uyt so get d and give the path of the remote folder the remote folder path was the remote folder and the local directory should be given so let's write local div uyt so i'm downloading all the contents of the remote folder into local directory called uit which is in the same folder as index.phi so let's try to run this file and here you can see all the contents of the remote folder are downloaded into my local folder called uit so using get d you can easily download all the remote folder files in the local folder the same way get r is a recursive get where you can even download all the subfolders of the remote folder also with your local folder so let's try to use get r and run this here now you can see the difference in get r the whole remote folder along with the folder will be downloaded into local path that's why i got the remote folder also the main folder also is been downloaded to uit but if i just use get d only the contents of the remote folder will be downloaded into my local folder so this is the difference between get r and get d when using get r the folder will be copied and while using get d the folder contents will be copied and using get r you can even download the subfolders of the remote folder all right let's try to get the metadata of a remote file using the stat function so if you call sftp.stat and give the file name path you can get the statistics of the file like what is the size of the file what is the mode of the file what is the timestamp of the recent access or modification time or the user id of the owner group id of the owner something like that so if i want to get the statistics of this file called xyz.txt in this remote folder i have to write sftp dot stat and i have to be the file path of the remote folder remote folder xyz dot txt and let me try to save it in a variable called x x equal to sftb stat and let's print x so i'll just save it and run this and you can see the statistics is printed in a single line but you can even access the statistics individually also by using something like x dot st underscore m time so it's like the modification time of the file so let's save it and run this it's a unix epoch times time you can convert it to a date time using this dd.datetime.fromtimestamp so i'll just write it here so to use dd i have to use datetime so i'll just write import datetime as dt so i'm converting this unix epoch timestamp into a python datetime using dt.daytime.fromtimestamp so let's save it and run this and here you can see this is the file modification time and i can see the file last action access time using s3 underscore a time so let's run this this is the last file access time and i can get the file size in bytes so i'll just remove this date time thing and write x dot st and the score size is the size of the file in bytes so let's run this it is six bytes the file is six bytes so this way using the sftp dot stack you can get the statistics of a single remote folder file the next thing is delete a remote folder so if you use sftp dot rmd you can delete a remote folder but the remote folder should be empty if the remote folder is not empty you'll get an error so let's try to use that function now so i'll just write sftp dot rm dill that is remove directory and here i'm going to write the remote folder path so i'll just save it and run this and here you got the io error because the remote folder is not empty so let me try to create a remote folder which is empty here i'll just create a new folder new folder fgd so i got a new remote folder called fgd and this is empty so let me delete that folder now fgd so let's save it and run this file again and here you can see my remote folder will be deleted so deleting a remote folder is really straightforward in psftp but deleting a remote folder which has contents requires some exercise to be done on our end itself so i've given a code to delete the remote folder with contents inside it the code is really simple so just find all the folder contents using the sftp.listing and you delete each file and then you delete the main folder and if you have subfolders you do the nesting so that's why we have done the recursive call so if you just use this function you can delete a remote folder also which has contents inside it let's try to create a non-empty remote folder so i just create a new directory here new directory let me name the directory as ghy so i've got a new remote directory let me create a new file here so that it's not empty i'll just create a new file and name it something like this so now i've got a non empty remote folder called ghy and it has one file in it and if i run using the normal sftp dot rm dill it will throw an error so we will use our function which you have created in this blog post so let's copy this function and paste it in my code so here i'm using the os module to join the file path and i'm using the start module to know whether the file path is actually a folder or a file all right now we've got our function and let's use that function to delete the non-empty remote folder so here i'm going to write rm the name of the function which you just created rm and it takes the sftp object as the first argument and the second argument is the path of the remote folder and the path of the remote folder was gh while so i'm going to save it and run this now now let's try to see whether the folder exists or not i'll just refresh it here and you can see the remote folder which was not empty is also deleted by using the function called rm which you have created right now so this way you can even remove non-empty remote server folders all right let's get to know about some other important sftp actions which we can do using pi s ftp python module sftp dot make this is a really useful function where you can create a new folder in the sftp server and you can even create the nested folders if they are not present so let's try to use this function now so if you want to create a folder inside this sftp server something like abc slash def that means you need a nested folder but there is no folder called abc make this function will actually create the intermediate folders also for you so let me try to use sftp dot make divs and here i let the folder path as abc slash def that means i have to create a new folder called df inside a folder called abc and if this abc folder is not present even it will be created by the sftp so i'll just save it and run this and let me to refresh it here here you can see abc and df so you are able to create nested folder and interpreted folders if they are not present are also created by pi sftp and you already covered sftp.stat and if you use the sd underscore size you can get the size of the remote file in bytes and sftp dot rename is also very useful function where you can rename a remote file or folder so let's try to use it now i have a file called abcd.txt let's try to rename it to xyz.txt so you just write sftp dot rename abcd.txt this is the original file name if it's present in a folder you can write folder one slash abcd.txt something like that and then you write the rename file name so i can write xyz dot txt so i am renaming the abcd.txt to xyz.txt i'll just save it and run this and i'll go to the remote server and just refresh it and here you can see i've renamed it to xyz.txt the next important thing is sftb.remove you can remove a single sftp file using the sftp dot remove function so let's try to use it now i got a file called xyz.txt and if i want to remove this file i can just write sftp dot remove sftp dot remove and i'll write xyz dot txt i'll save it and run this and if i go to this remote server and refresh you can see i don't have xyz.txt and that's it guys that was all about using pi s ftp python module to perform some important actions on an sftp remote server using your python code i have created a blog post on using 50p python module for sftp server interaction i have also given you all the sample codes so you can copy paste and practice it to your own computer so be sure to check out the link of this blog post in the description of this video in the references section i given the link of the pi sftp cookbook which is a really great resource to learn bias ftp very fast here they have given sample codes and you can learn each function using a sample code for example if i'm using the sftp dot get here you can see sftp dot get and he's telling how this function can be used so this cookbook is really useful for you to understand each function of the pi sftp module and if you don't find your required function in the 50p cookbook you go to the home and click on the api tab and here you can see all the functions will be given along with the parameters description and one more thing i want to highlight here is pios50p is just a wrapper around the parameco python module which actually implements the sftp client so this is a link for the parameter docs please leave your valuable feedback or any questions in the comment section hope you like this video guys thank you for watching

Channel: Learning Software

Views: 22,138

Keywords: tutorial, taming_python, taming_python_skill, sftp, openssh, pysftp, paramiko

Id: IQh0K_6ecrU

Length: 24min 29sec (1469 seconds)

Published: Sun Apr 03 2022