How to Install PySpark on Linux: Step-by-Step Guide for Developers

Setting up Python and PySpark correctly on a Linux system is one of the most common challenges developers face when starting with data engineering or distributed processing. Whether you are preparing for Databricks, working with local Spark jobs, or experimenting with large datasets, having a clean and reliable setup is critical.

This guide walks you through a step-by-step installation of Python and PySpark on Linux, ensuring your environment is production-ready and aligned with real-world workflows.

Step-by-Step Guide to Install PySpark on Linux Environment

Step 1: Install Python on Linux

Most Linux distributions come with Python pre-installed, but it is recommended to install a stable version manually.
Check existing version:

python3 --version

If not installed or outdated:

For Ubuntu/Debian:

sudo apt update
sudo apt install python3 python3-pip -y

For CentOS/RHEL:

sudo yum install python3 python3-pip -y

Verify installation:

python3 --version
pip3 --version

Step 2: Install Java (Required for PySpark)

PySpark depends on Java (JDK).

Install OpenJDK:

sudo apt install openjdk-11-jdk -y

Set JAVA_HOME:

export JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64
export PATH=$PATH:$JAVA_HOME/bin

Verify:

java -version

Step 3: Install PySpark

Now install pyspark using pip:

pip3 install pyspark

This is the easiest method for beginners and works well for local setups.

Alternatively, you can explore the official pyspark github repository for advanced configurations and source-based installations.

Step 4: Verify PySpark Installation

Run Python shell:

python3

Then:

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("Test").getOrCreate()
spark.range(5).show()

If this runs successfully, your pyspark download and setup is complete.

Step 5: (Optional) Install Apache Spark Manually

If you want more control:

Download Spark:

wget https://downloads.apache.org/spark/spark-3.5.0/spark-3.5.0-bin-hadoop3.tgz

Extract:

tar -xvzf spark-3.5.0-bin-hadoop3.tgz

Set environment variables:

export SPARK_HOME=~/spark-3.5.0-bin-hadoop3
export PATH=$PATH:$SPARK_HOME/bin

This method is useful when working with custom Spark configurations or Databricks-like environments.

Step 6: Configure PySpark for Development

Set Python for Spark:

export PYSPARK_PYTHON=python3
export PYSPARK_DRIVER_PYTHON=python3

Run:

pyspark

This opens an interactive shell similar to what you experience in Databricks notebooks.

Why Proper Setup Matters

Incorrect setup is one of the biggest reasons developers face issues like:

PySpark not starting
Java gateway errors
Version mismatch problems
Performance issues

A clean installation ensures your environment behaves consistently across local, cloud, and production systems.

Practical Insight

If you’re planning to work on real-world data pipelines or migrate workflows to Databricks, understanding this setup deeply helps avoid common pitfalls later. Most performance and debugging issues in Spark originate from environment misconfiguration rather than code.

If you get stuck while setting up or configuring pyspark, you can get direct help through a focused session by TheCodeWizard.

This approach ensures your python programming and pyspark environment is not just installed—but actually usable for real development and scalable workloads.