Install and setup/configure Apache PySpark 3.5.1 on Windows 11 |Install PySpark on Windows 11 | 2024

hello guys in this video we will see how to install and configure Apache Spar 3.5.1 on Windows 11 operating system it requires V or any software which is required to unip then Java Python and WIS or Hardo installation okay I will show you all one by one so first install Java I have kept the link video link in the video description to download and install Java on Windows 11 then python download python it is simple okay see the latest one is click on downloads then download this python 3.12 do3 okay so it is simple just download it and go to downloads you have to just double click on this one it is very simple the next one is vils you can get the UNS from the following GitHub link see see here we have the versions of Hado okay I will show you which version we have to download go to spock dot Apachi Spock do api. orc click on download I will keep this link in the video description okay then see here the latest version is 3.5.1 so we are going to download and install this version then choose a package type see here we have the free built for Apache how 3.3 or later okay but we don't have the older version so we have to download 3.3 or later okay go to here in this GitHub account then select 3. 3.0 or 3.31 bin both are fine Okay click on 3.30 then click on this win.exe click on this download button see download started see we have successfully downloaded next we have to download the thisp J file click on it click on this first link see download started see it size is 382 megab wait until this download completes see download is completed the free requisit we have downloaded and now we will see how to configure see I have the wiip or Windra see I have Windra software the first one matched the second one is open CMD type Java hyphen hyphen version see open jdk Java 17 is installed and the second one is python type Python 3 and hit enter is python see we have the python latest version that is 3.12 do3 okay this is the version that I have installed and next one is inits okay now we will see how to configure fre match go to C drive first create a directory for Hardo okay make the folder name as Hardo hit enter in this folder create one more directory name it as bin hit enter go to the drive go to downloads directory copy vs copy it paste here then next create one more directory for Spock click on new create directory spark hit enter now go to again downloads directory right click on the spark bin G file Windra then use Windra or any software and extract files then give the spark directory click on okay see extraction is completed go to spark then copy all these files cut right click cut them go to spark directory paste it remove this empty directory all data is ready so what we have done until now Java is installed python is installed spark extracted and Hardo directory created in that bin is there okay so next we have to configure the environment variables in Windows 11 right click on this machine this PC then click on properties then advanced settings then environment variables otherwise start then search for environment edit environment variables click on see we got the same window click on this also we got the same Okay click on environment variables see I have already set the Java home that how to set Java home I have that is mentioned in the video see here just we have to give the Java _ home then we have to give the path where Java is installed then the path variable CLI on path edit here at the bottom we have to create new variables that I will show you first create a new direct new variable for Hardo click on new then provide name as hore home then in variable value click on browse then go to the drive where we have kept that in for file that it see select only this hard directory not the pin click on okay now select the path and click on edit then click on new then give the percentile I will keep all the directories and the environment variables values also you hard underscore home percentile back slash then give the bin click on okay now next create a new system variable on new then give spark home then click on browse go to C drive go to spark select Spark select spark then click on Okay click on okay then click on path again click on edit new then percentile spark uncore home spark binc back slash bin then click on okay next next we have to see I have the Java home variable even if you don't know just go to the where Java is installed you have the path like this you have to copy this path then you have to create new then paste the path then Java uncore home like this we have to give okay I have different version all to see I have already set the Java if you installed Java 17 that see I have made a video how to download and install Java 177 on windows so you can go through it and you can install it okay then click on see I have set the Java environment variable then select the path click on edit new then give the passent Java uncore home personent back slash bin then click on Okay click on Okay click on okay all set now open command prompt CMD then give the spark hyphen it is hyphen or minus it is not underscore then shell see you haven't received error which means it is Success see we have successfully connected okay and this is the see this is the web UI okay copy it just select it and hit enter then go to the this paste it see this is the we you have the stages storage environment executors and and many okay and jobs okay this is Scala contrl D to quit see we have connected spark P spark Scala version but and see here this is the version 3.5.1 and this is the Java version okay so if you want to connect to P give the P spark no need to give anything just give the pice spark and hit enter see we have successfully connected see this is see this is the web UI so copy it hit enter paste it it enter see we got the same one this is Spice Box shell we can run any commands here we can create the rddr we can perform any actions okay so in this video we have seen how to download and install Apachi spark for python R SCA on Windows 11 operating system for more P spark Scala videos please subscribe my channel thank you

Channel: r2schools

Views: 598

Keywords: Apache Spark 3.5.1 setup on Windows 11, Apache Spark for Scala setup on Windows, Apache Spark for Scala setup on Windows 11, Apache Pyspark setup on Windows 11, Apache Pyspark setup on Windows, Apache Spark Installation on Windows, Apache Spark Installation on Windows 11, Apache Spark 3.5 Installation on Windows 11, Apache Spark 3.5.1 Installation on Windows 11, Apache Spark 3.3 Installation on Windows 11, Apache Spark 3 Installation on Windows, PySpark Tutorials, Scala Tutorials

Id: rYY0LFdmI8s

Length: 10min 18sec (618 seconds)

Published: Tue Apr 30 2024