Upload A DataFrame To Google Cloud Storage As A CSV File Without Saving Locally Using Python

hey how's it going guys in this tutorial we'll learn how to upload a pendant snare frame to a google cloud storage in python while saving the table to a local drive so as i'm transitioning to do more data engineering work i noticed the need to rely on cloud storage services to host their files and one of the common tasks is to upload the iphone table to a cloud storage directly all right so let's say every month i have a task which is to export tesla's stock historical price to google cloud storage now here in my sql for bigquery project in my cloud storage service i have this bucket called reddit expose now let's say one the workflow is to uh export the historical price data into a data fan then from there we're going to export the data frame as a csv file and load the csv file to this reader expose bucket now let's take a look at how we can do that in python so first i'm going to import the libraries that i'll be using for this size so if we look at the import statement from the standard module i'm going to import date time and time modules and here's the pandas library importing statement and to work with google cloud storage i'll be using google cloud storage api and before you can use google cloud storage api make sure that you have enabled the google cloud storage api service first and if you don't have the library installed you can use the command pip install google cloud storage now let's move on to the next step so here i have a code plot which is to load the historical price from yahoo swap site and load the table using pd.vcsv then i'll pass the url now if i go ahead and run this code block and if i print the dataframe object and it's going to give us the historical price table for july 1st all the way to july 29th since july 30 and 31st are on the weekend and it's going to be a step one load the data into a data frame right so for step two we're going to load the table here to our cloud storage service all right so it's going to be load table actually uh should we should be uploading that low upload the airframe table to a cloud storage service now using google cloud 3 service so here i'm going to create an instance of the google cloud storage client instance and i'll name the instance client next we need to create entirety that point to the bucket so i'm going to reference the client's object the git bucket and followed by the box name now name the outputs let's name this as reader actually let me name this to export bucket and to upload the data frame object to my reader exposed bucket so here it actually maximize the window so here let's look at line 17 here all right so let's break down the statement here so from the data frame object if we run the to csv method while providing a file path it's going to return a single string and that's what the csv file content looks like and we fix the option name and this should be export bucket now i found the export barclay object so this is referencing the bug itself in form to upload a file we need to use the blog method followed by providing the option name so this is going to be bucket the blob and it's going to be a file name diablo so uh this statement here is going to create the uh the object name right so let me print the file name first should we go here all right so if i upload the file the app java name is going to be tesla followed by today's timestamp if i grab this stem in here and if i print the output object type oh i forgot to create the bucket first right so if i print the output object type and because we're using the block method to first of all create the linkage to point to where we want to save the file this is going to return us a storage.blob dot blob object and from the blog object itself the symmetrical upload from string so we can use this method to provide the string directly as well as providing the file type to allow the upload from string method to figure out what type file we're trying to upload now let me go ahead and terminate this session all right so if i go ahead and run the python script from scratch all right so it looks like the script is complete now if i go back to my cloud storage and if i refresh the page and here's the test last historical price csv file all right so this is going to be everything i'm going to show in this video and hopefully you guys found this video useful and as always thank you guys for watching i'll see you guys in the next video

Channel: Jie Jenn

Views: 7,550

Keywords: google cloud storage, google cloud, gcs, google cloud storage python

Id: IHUJ3g01xmI

Length: 6min 34sec (394 seconds)

Published: Tue Sep 27 2022