How to Install Spark | Pyspark | Python | Pycharm IDE on Local Machine

welcome in this video we will see how to install a Hadoop framework and Pi spark on the local machine so for this I am going to use a pycharm ID so I don't have any ID installed right now so I will first of all install pycharm ID so to install pythm ID you can go to Google and you can just type Community Edition pycharm download so this is the first link that you get you can click on that and here there is a Community Edition setup so you can click on this download button and the setup will get downloaded I have already downloaded the setup so I will not click on this but you can download it by clicking on this button now once the setup is downloaded you will get such kind of a file so first of all we have to install this so just double click on this so after that you will get this window so you have to click on next then again click on next then don't forget to click on this that is update path variable so just click on this and click on next next and install so this will take some time so now my installation is complete so now I'm getting this message whether I want to reboot now or reboot manually later on so it's better to reboot the computer once the installation is finished so you can click on reboot now and you can click on finish now next is you have to download the spark Hadoop framework so for that again you can Google it spark Hadoop download then you can go to this first link now you can click on this download spark link and now you can click on this first link so it will start downloading the spark Auto so I have already downloaded this spark Hadoop framework then the next thing that is required is the win utils so for that again you can Google it so you can type win utils download so you'll get this link so just click on that now in that again you can go to the latest version that is Hadoop 3.0.0 now here you will get that win utils file so just click on that file and then you can click on download so that file will get downloaded next is you you need to download the Java jdk so just again Google it as jdk downloads so you'll get this link so click on that link now here you can click on Windows and you can download any of these links so I have already downloaded so you can click on this link and download the jdk now once all these files are downloaded next is we will install the Java so this is the Java jdk setup file so just double click on this then click on next next so it will install the jdk now after in installation you can click on close now next is you can go to C drive and you can create one folder named spark now inside this folder you can extract the spark framework that we have downloaded so this will take some time so now the file contents are extracted over here now again go to C drive and create one more folder named as Hadoop now inside this folder we have to create one more folder named as bin inside this folder we need to copy the win utils file that we have downloaded now you can go to the system environment variables so for that we can click on the Windows button and type environment variable so you can click on this environment variables now in this we have to create a system variables so just click on new now in variable names you can type Hadoop underscore home all in capitals and in variable value you can give the path of the Hadoop filter that we have created in the C drive after that click on OK after that again click on new now this time give the name as spark underscore home and the variable value will be the folder where you extracted the spark framework so this is the path then click on OK then again click on new this time give the name as Java underscore home and give the path where you have installed the Java so you can go to C drive so here I have the Java folder inside this we have jdk 20 so you can copy this path and paste it over here and click on OK now again you can go to this user variables so just click on path and click on edit and you can add A New Path here you can type percent spark underscore home percent slash bill then again click on new type the second path as percent Hadoop underscore home percent slash bin then again click on new type the third path as person spark underscore home percent slash python then click on new again and type the fourth path as percent python path percent now next is again we have to go to C drive where we have installed that Spark so go inside this folder in this we are having python and in this we are having Library folder so you have to select this path so basically you have to select the path till this ZIP file so this path along with this ZIP file name so I will first of all copy copy this path then click on new and paste that path over here then click on slash and copy this remaining path so you can click on edit and you can paste that path over here and don't forget the extension that is dot zip okay then after this you can click on OK then click on OK click on OK now open the command prompt and you can type here a command Spark iPhone submit space hyphen iPhone version and hit enter so if the spark framework is successfully installed then you'll get such kind of output now after that you have to restart the system now we need to configure the pi charm ID to use Pi spark inside pycharm ID so for that open the pycharm that we have installed so when you open it for the first time it will ask you to select and accept the agreement so click on continue now here you can click on new project you can give some name to your project I will keep it as it is then click on create so once the project is created it will open this window now in this you can click on this settings button which is available on the right side top corner then in that you can go to settings so here you can go to this project and you can click on this python interpreter so you will get what are the packages already installed then you can click on this project structure and on this right side here you have an option to add content root so click on this and now we have to go to the C drive where we have installed that Pi spark framework so we have created one folder named Spark so this is that folder so inside this we have to go to python inside that we have to go to live and we have to select these both files and click on OK so once these files are added over here then you can click on apply and you can click on OK now we need to check whether these Pi spark is getting executed in this pycharm ID or not so for that you can paste the code over here so using this code we can create a session spark session so if I run this so you can click on this allow access from the Windows firewall so you can see that here I have got the messages process finished with exit code 0 it means that there are no errors and my spark session is successfully created it means that spark is properly installed now I can read one file CSV file into the spark data frame for that I can use this code so I have one file named as employees.csv I will read that into a spark data frame named as DF and will print that schema of that file so if I run this file all right so now I've got the output so this is nothing but the schema of that particular file so my spark is properly installed and I can make use of pycharm ID to write the pi spark code so in the next video we will see how to install Pi spark in jupyter Notebook the detailed steps that are involved in installation of Pi spark are available in the description of this video I hope you have liked this video so please don't forget to like share and subscribe to my channel thank you

Channel: Studytronix

Views: 12,032

Keywords: #pyspark, #pycharm, #spark, #installation, #local, #machine

Id: 6Cn_Gb0RMG8

Length: 10min 54sec (654 seconds)

Published: Wed May 03 2023