How To Install Spark Server In Windows

Apache Spark is a fast and full general-purpose cluster computing system. It provides high-level APIs in Java, Scala, Python and R, and an optimized engine that supports full general execution graphs. It also supports a rich set of higher-level tools including Spark SQL for SQL and structured information processing, MLlib for machine learning, GraphX for graph processing, and Spark Streaming.

In this document, we will encompass the installation procedure of Apache Spark on Windows 10 operating arrangement

Prerequisites

This guide assumes that you lot are using Windows 10 and the user had admin permissions.

Arrangement requirements:

• Windows 10 OS
• At least 4 GB RAM
• Free space of at least 20 GB

Installation Procedure

Step ane: Become to the beneath official download page of Apache Spark and choose the latest release. For the package type, choose 'Pre-built for Apache Hadoop'.

The page will expect like beneath.

Step 2:   One time the download is completed unzip the file, to unzip the file using WinZip or WinRAR or 7-ZIP.

Pace 3: Create a folder chosen Spark under your user Directory like below and copy paste the content from the unzipped file.

C:\Users\<USER>\Spark

Information technology looks like beneath after copy-pasting into the Spark directory.

Pace iv: Go to the conf folder and open log file chosen, log4j.properties. template. Change INFO to WARN (It tin can be Fault to reduce the log). This and next steps are optional.

Remove. template so that Spark can read the file.

Before removing. template all files await similar below.

Later removing. template extension, files will look similar below

Pace v: Now we need to configure path.

Go to Command Console -> System and Security -> System -> Advanced Settings -> Environment Variables

Add below new user variable (or Organisation variable) (To add together new user variable click on New button under User variable for <USER>)

Click OK.

Add %SPARK_HOME%\bin to the path variable.

Click OK.

Stride half dozen: Spark needs a piece of Hadoop to run. For Hadoop two.7, you need to install winutils.exe.

Yous can detect winutils.exe from below page

Download it.

Pace seven: Create a folder called winutils in C bulldoze and create a folder called bin inside. Then, motion the downloaded winutils file to the bin folder.

C:\winutils\bin

Add the user (or system) variable %HADOOP_HOME% like SPARK_HOME.

Click OK.

Stride viii : To install Apache Spark, Java should be installed on your computer. If you lot don't have java installed in your system. Please follow the below procedure

Java Installation Steps :

• Go to the official Java site mentioned beneath  the folio.

Accept Licence Understanding for Java SE Development Kit 8u201

• Download jdk-8u201-windows-x64.exe file
• Double Click on Downloaded .exe file, you lot will the window shown below.

• ClickNext.
• And so below window will be displayed.

• Click Next.
• Below window volition exist displayed after some process.

• Click Close.

Exam Java Installation :

Open Command Line and type java -version,then information technology should brandish installed version of Java

Y'all should as well check JAVA_HOME and path of %JAVA_HOME%\bin included in user variables (or system variables)

i. In the terminate, the environment variables accept three new paths (if you demand to add Java path, otherwise SPARK_HOME and HADOOP_HOME).

2. Create c:\tmp\hive directory. This step is not necessary for later versions of Spark. When y'all start start Spark, it creates the binder by itself. However, it is the best practice to create a folder.

C:\tmp\hive

Test Installation :

Open command line and type spark-crush, y'all get the issue equally below.

Nosotros accept completed spark installation on Windows system. Allow's create RDD and     Data frame

We create 1 RDD and Data frame and so will end up.

1. Nosotros tin create RDD in three means, we will use one way to create RDD.

Define whatever list then parallelize it. It will create RDD. Below is code and copy paste it one past one on the control line.

val listing = Assortment(1,ii,3,4,five) val rdd = sc.parallelize(list)

Above volition create RDD.

2. At present we volition create a Data frame from RDD. Follow the beneath steps to create Dataframe.

import spark.implicits._ val df = rdd.toDF("id")

Above lawmaking will create Dataframe with id as a cavalcade.

To display the data in Dataframe use below command.

Df.prove()

Information technology will display the below output.

How to uninstall Spark from Windows x System:

Please follow below steps to uninstall spark on Windows 10.

1. Remove below System/User variables from the arrangement.
2. SPARK_HOME
3. HADOOP_HOME

To remove Organization/User variables please follow below steps:

Become to Control Panel -> System and Security -> Organization -> Advanced Settings -> Environment Variables, then observe SPARK_HOME and HADOOP_HOME and then select them, and press DELETE button.

Find Path variable Edit -> Select %SPARK_HOME%\bin -> Press DELETE Button

Select % HADOOP_HOME%\bin -> Press DELETE Button -> OK Push button

Open up Command Prompt the type spark-shell then enter, now nosotros get an error. Now we tin confirm that Spark is successfully uninstalled from the System.

Source: https://www.knowledgehut.com/blog/big-data/how-to-install-apache-spark-on-windows

Posted by: shanerloadere1987.blogspot.com

Share this post