11. withColumnRenamed() usage in PySpark | Azure Databricks | Azure Synapse Analytics

hi friends welcome to first release YouTube channel this is part 11 in pi spark playlist in this video we are going to discuss about with column renamed function usage in pi spark this function actually helps you to rename any column that means existing column in data frame so please watch my previous videos in the pi spark playlist because all the videos are in a sequence order in our last video we discussed about usage of with column function that will be used to add a new column or to change the existing column data type are the existing column values but this function here helps you to change the column name in the data frame that means the existing column you can rename it so let me show you how to do that so let me go to browser and I have already opened my data databricks workspace here so let's go to workspace let me go to users under my user account let me try to create a new notebook and let me name it like with column renamed notebook so this is The Notebook name python is a default language let me hit create button to create the notebook here so let me close this dialog here and here let's try to declare a variable called Data and inside this data variable let's try to pass a list list of tuples in the first two pull one and Mahir and then let's use some salary also here maybe like 2000 okay and then in the second Tuple let's use two comma so whatever these two pull values these are going to act as a rows so you know this if you have seen my videos from starting you know to you know how to create a data frame with this hard-coded values right so I am doing the same thing also let's try to create another variable called The Columns it is going to be a list so ID column then name column then salary column okay so now let's try to use this spark object that is nothing but like a spark fashion object on top of that we have create data frame function to this data function there is a parameter called Data to the data parameter parameter let's pass this data variable and then there is a parameter called schema to the schema parameter let's pass these columns so this entire code is going to create a data frame so let me declare a variable called TF to store the data frame now let me hit DF dot show to to see this data frame in a tabular format here so I am executing the cell here now let's wait for the command to execute here command executed successfully I can see ID column name column salary column which is the data as well so this is clear cool so now what if I want to rename this salary column to salary amount something like that how to do that to do that we have to use with column renamed function so data frames are actually immutable that means you cannot change the existing data frame so always whatever the transformation function you apply it will actually generate another data frame from it and that data frame you can store it into some variable so here if I say DF Dot with column with the column rename right so this is the function we have to use and here right so this function if you see it will take two parameters so let me do one thing so d f dot here let's use a help function to the help function d f Dot with the column one second sorry so let me remove this entire code to the help function d f dot control space with column renamed and let me execute this to see the documentation of with column renewable function if you see this will generate a new data frame and give it back and it will actually rename the existing column and if you see here it takes two parameters existing column name and a new column name both the parameters should be in a string format and you have example also here it is renaming H column to H2 so let me try to do the same thing here so in this code here we have a salary column so existing column name is salary this column I want to convert it into salary underscore amount maybe so let's assume now this entire code actually generates a new data frame it will not change the existing data frame why because existing data frames are the data frames whatever you have they are immutable you cannot change them so always any transformation you apply it will actually generate a new data frame to prove that I am not storing this entire code into another variable so let it be on the same data frame I am using with the column renamed function and I am simply doing the data frame dot show so now let's see what will happen so before that let me close this cell here whatever I opened here to see the documentation and here let me go back to my first cell and let me shift enter and see what will happen command is running now you still see sorry column why because I am showing this data frame whatever we created here on top of that data frame we applied the transformation function but still as I said these functions are going to generate a new data frame they are not going to change the existing data Frame data frames are immutable you cannot change it so to make you the things sense let's use another variable called df1 and let's try to store the data frame whatever it gets create here into this df1 and now let's try to use df1.0 and now if I hit shift enter you will be seeing see cell there is no salary column now it now it can become like salary amount column right why that happened in this data frame we took the salary column and we renamed it with this new column name and that entire code will generate another data frame so that data frame we took it here and that new data frame we are displaying it here using the show function so this is a using the width column renamed function you can actually rename the existing columns so it is the same code whatever I have shown here we have a three columns so we used width column renamed function and we are renaming salary column to salary amount and it gives a new data frame and that data frame we are showing it here so that's it in this video thank you for watching please subscribe to my channel and press the Bell icon to get the notification whenever videos thank you so much

Channel: WafaStudies

Views: 14,006

Keywords: PySpark for beginners, PySpark Playlist, PySpark Videos, Learn PySpark, PySpark for data engineers, dataengineers PySpark, PySpark in Azure Synapse Analytics, PySpark in Azure databricks, Understand PySpark, What is PySpark, PySpark in simple explaination, PySpark Overview, synapse pyspark, spark, withColumnRenamed() usage in pyspark, df.withColumnRenamed() in pyspark, change existing column name in dataframe, modify column name in dataframe in azure databricks, pyspark

Id: z2_ajv_aY2Y

Length: 6min 38sec (398 seconds)

Published: Wed Nov 02 2022