Get Spark and Python running on your computer

What are we installing, and why? 🧩

Spark is the program that splits a big job up across lots of computers — the same trick as getting every classroom to count their own results instead of one poor person counting the whole school district. To run it on one computer we need three things:

• Python 🐍 — the language we write our programs in. We'll install it with conda, which keeps this project's Python in its own tidy box, separate from anything else on the computer.
• Java ☕ — Spark itself is written in Java and Scala, so even though we type Python, Java has to be there underneath. This is the step people forget most often, and it causes the most confusing error messages.
• PySpark and JupyterLab — PySpark is the Python flavour of Spark, and JupyterLab is the notebook where we type code and see answers appear. We use PySpark 4.2 throughout the book, so that's the version we ask for.

Plan for about 20–40 minutes, most of it waiting for downloads. You'll need a Mac, Windows, or Linux computer with roughly 5 GB of free disk space and an internet connection.

Pick your computer 👇

Then everybody meets back up at testing that it worked and starting the notebook.

🍎 Instructions for Mac

Open the Terminal app: press ⌘+Space, type "Terminal", press Enter. Commands go in that black-and-white window.

Step 1: install conda

We recommend Miniforge: a small, community-run conda that's free for everybody, including at work. (The better-known Anaconda installer now needs a paid licence at larger organisations. Miniforge skips that entire conversation.)

If you already have Homebrew (type brew --version to find out), it's two lines — the second one wires conda into your shell, which the first one doesn't do on its own:

brew install --cask miniforge
conda init zsh

Then close Terminal and open it again. You'll know it worked when the prompt has (base) at the front, like (base) yourname@laptop ~ % — and you can jump straight to Step 2.

If you don't have Homebrew, that's fine — you don't need it. Run these two lines instead. They work out which Mac you have on their own, so there's nothing to choose:

curl -L -O "https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-$(uname)-$(uname -m).sh"
bash Miniforge3-$(uname)-$(uname -m).sh

That second line starts an installer which asks you three things, and this is where everybody gets stuck:

1. "Please, press ENTER to continue" — press Enter.
2. A huge wall of legal text scrolls past. Tap the space bar to page through it, or press q to jump straight to the end. It is very long. It's supposed to be.
3. "Do you accept the license terms?" — type the whole word yes and press Enter. Just y won't do for this one. Then it asks whether to initialize conda: yes again, Enter.

When it finishes, close Terminal and open it again — conda isn't switched on until you do. You'll know it worked when the prompt has (base) at the front, like (base) yourname@laptop ~ %.

Step 2: make a home for the project

This creates a little sandbox called dc4k with its own Python and its own Java, so nothing we do here can break anything else on the computer.

conda create -n dc4k -c conda-forge python=3.12 openjdk=21
conda activate dc4k

It lists a screenful of packages and then asks Proceed ([y]/n)? — type the letter y and press Enter. (Anything in a terminal offering you [y]/n works this way: one letter, then Enter. The one in square brackets is what you get if you just press Enter without typing anything.) Downloading takes a few minutes. When your prompt comes back it should say (dc4k) at the front instead of (base).

Step 3: install PySpark and the notebook

pip install --upgrade "pyspark[sql]==4.2.*" jupyterlab pandas

That installs PySpark 4.2. It's a biggish download — a few hundred megabytes — so this is an excellent moment to go get a snack. 🍪 It scrolls a lot of text, and it may sit completely still for a minute at a time. That's normal, so don't press anything. You'll know it finished when you see a line starting Successfully installed and your prompt comes back.

Now skip down to testing that it worked.

python3 -m venv ~/dc4k
source ~/dc4k/bin/activate
pip install --upgrade "pyspark[sql]==4.2.*" jupyterlab pandas

🐧 Instructions for Linux

Open a terminal — on most systems Ctrl+Alt+T does it. If that does nothing, search your applications for "Terminal" or "Console".

Step 1: install conda

curl -L -O "https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-$(uname)-$(uname -m).sh"
bash Miniforge3-$(uname)-$(uname -m).sh

Press Enter at the first prompt, then space-bar through the licence (or press q to skip to the end), then type the whole word yes to accept it and yes again when it offers to initialize conda. Then close the terminal and open a new one. The prompt should now start with (base).

Step 2: make a home for the project

conda create -n dc4k -c conda-forge python=3.12 openjdk=21
conda activate dc4k

Type y and press Enter when it asks Proceed ([y]/n)?. When the prompt comes back it changes from (base) to (dc4k). You'll need to type conda activate dc4k in every new terminal window from now on.

Step 3: install PySpark and the notebook

pip install --upgrade "pyspark[sql]==4.2.*" jupyterlab pandas

Wait for Successfully installed and your prompt to come back.

python3 -m venv ~/dc4k
source ~/dc4k/bin/activate
pip install --upgrade "pyspark[sql]==4.2.*" jupyterlab pandas

🪟 Instructions for Windows

Step 1: install conda

Download the Miniforge installer for Windows from conda-forge.org/download and run it. Accept the defaults, and install it "Just Me" into a path with no spaces in it — the default, C:\Users\yourname\miniforge3, is perfect. Spaces in paths confuse Spark.

When it's done, open Miniforge Prompt from the Start menu. That's the window you'll use for everything below — not the regular Command Prompt or PowerShell, because conda is only switched on inside this one.

Step 2: make a home for the project

conda create -n dc4k -c conda-forge python=3.12 openjdk=21
conda activate dc4k

Type y and press Enter when it asks Proceed ([y]/n)?. The prompt should change to say (dc4k).

Step 3: install PySpark and the notebook

pip install --upgrade "pyspark[sql]==4.2.*" jupyterlab pandas

Wait for Successfully installed and your prompt to come back.

Step 4: the awkward Windows bit (winutils)

On Windows, Spark wants two little Hadoop helper files that it doesn't ship with, and nobody has ever fixed this properly — so we do it by hand. Without them Spark usually still starts, then prints a wall of red and refuses to save anything. Sorry. Here's how to give it what it wants:

1. Make a folder called C:\hadoop\bin.
2. Download winutils.exe and hadoop.dll from the cdarlint/winutils collection on GitHub. Pick the folder with the highest version number available, open its bin folder, then click each file and press the Download raw file button — the small downward arrow near the top right. GitHub shows you a preview page first, so don't worry when the first click doesn't seem to download anything. Windows may also warn you about downloading an .exe; choose Keep.
3. Put both files in C:\hadoop\bin.
4. Tell Windows where they are. This step sounds alarming because Windows calls it "system settings", but all you're doing is writing down a folder name so Spark can find it — you can delete it again later and nothing will mind. Press Start, type "environment variables", and choose Edit the system environment variables → Environment Variables. Under "User variables" click New and add a variable named HADOOP_HOME with the value C:\hadoop. Then find Path in that same list, click Edit → New, and add C:\hadoop\bin.
5. Close the Miniforge Prompt and open a new one so it notices the change. Don't forget conda activate dc4k again.

Testing that it worked (everyone) 🧪

In your terminal, with (dc4k) showing in the prompt, run:

pyspark --version

You should see the word Spark drawn out big using slashes and underscores (that's Spark showing off), a version number starting with 4.2, and a line naming the Java it found. If you see that, Java and Spark have found each other and the hard part is over. 🎉

Now let's actually compute something. Start the interactive shell:

pyspark

Wait for the >>> prompt. It can take 20 seconds and print some warnings along the way — warnings are fine, that's just Spark talking to itself. Just above the prompt it should say SparkSession available as 'spark', which is Spark handing you a ready-made variable called spark. Then run:

spark.range(1000000).filter("id % 7 == 0").count()

It should think for a moment and answer 142858.

Where does that number come from? There are 142,857 multiples of seven under a million — and 142857 × 7 = 999999, which is one of the tidiest facts in arithmetic — and then zero counts too, because zero divides by everything. So: 142,858.

Starting the notebook (everyone) 📓

The book uses notebooks, which are much nicer than the shell: you can edit, re-run, and keep your work. In the terminal:

conda activate dc4k
jupyter lab

Two things happen. First, the terminal fills with text and then stops giving you a prompt back. It isn't frozen — that's JupyterLab running, and it has to keep running the whole time you use the notebook, so leave that window open and alone. Second, your web browser opens with JupyterLab in it. (If no browser opens, find the line in the terminal starting http://localhost:8888/lab?token= and copy that whole line into your browser's address bar.)

Click Python 3 under "Notebook" to make a new one. A notebook is a stack of boxes called cells: you type code in a cell, then press Shift+Enter (hold Shift down, press Enter) to run it, and the answer appears underneath. Click the first cell, type this, and press Shift+Enter:

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("dc4k").getOrCreate()
spark.range(1000000).filter("id % 7 == 0").count()

You should get 142858 again, this time inside the notebook. That's the whole setup done — the rest of the book is the fun part.

When things go wrong 🔧

Almost everyone hits at least one of these. Nothing on this list means the computer is broken.

conda: command not found (or "not recognized")

The installer finished but the terminal hasn't noticed yet. Close every terminal window and open a fresh one. On Windows, use Miniforge Prompt from the Start menu, not Command Prompt or PowerShell.

No such file or directory right after downloading Miniforge

The file isn't named what the command expected. Run ls Miniforge3* (Mac and Linux) to see its real name, then use that name. If you downloaded it with a browser instead of the terminal, it's probably in your Downloads folder, so try ls ~/Downloads/Miniforge3*.

JAVA_HOME is not set, or Unable to locate a Java Runtime

Java isn't installed, or Spark can't see it. Inside your conda environment, run conda install -c conda-forge openjdk=21 and try again. Check what you've got with java -version — Spark 4.2 runs on Java 17, 21, or 25, and we install 21.

UnsupportedClassVersionError, or a complaint about "class file version"

The Java that's installed is too old for this version of Spark. Install a newer one: conda install -c conda-forge openjdk=21.

TypeError: 'JavaPackage' object is not callable

Almost always the same problem wearing a disguise — Python found PySpark but not a usable Java. Fix Java as above.

error: externally-managed-environment from pip

You're trying to install into the computer's own Python, which modern systems protect on purpose. Activate your conda environment first (conda activate dc4k), or make a virtual environment as described in the Linux section.

A very long red Py4JJavaError wall of text

Don't panic, and don't read all of it. Scroll to the top of the message: the first line or two is the real cause, and everything below it is Spark explaining where it was standing when things went wrong.

macOS: "Do you want the application to accept incoming network connections?"

Click Allow. Spark's pieces talk to each other over the network even when they're all on the one computer. Nothing is leaving your machine.

Windows: errors mentioning winutils, NativeIO, or UnsatisfiedLinkError

That's step 4. Check that both files really are in C:\hadoop\bin, that HADOOP_HOME is set to C:\hadoop (with no \bin on the end), and that you opened a brand new prompt afterwards. If it still complains, it's the Hadoop version mismatch described in step 4, and WSL is the way out.

Address already in use, or port 4040 is busy

An older Spark is still running somewhere. Close other notebooks and terminals — or just ignore it, because Spark will quietly move itself to port 4041.

The notebook can't find PySpark, even though you installed it

The notebook is probably running from a different Python. Quit it, run conda activate dc4k, and start jupyter lab again from that same window. Inside a notebook, import sys; print(sys.executable) should show a path with dc4k in it.

A complaint that PyArrow must be installed

Something in the book asked Spark to hand data to pandas, which needs an extra piece. Run pip install --upgrade "pyspark[sql]==4.2.*" — the [sql] part is what brings it along.

Everything is just… slow

On one computer, Spark spends a lot of effort organizing work that a plain Python loop would finish instantly. That's expected, and it's the whole point of the note near the top of this page. Spark's superpower only shows up when the data is bigger than one machine.

Nothing on this list helps

Then stop. Local Spark installs go wrong in ways nobody has catalogued yet, and losing a whole evening to one is a bad trade. Use a cloud notebook instead — the code in the book is identical either way, and it's about ten minutes of clicking.

Starting up next time ⏭️

Good news: once it's installed, getting going again is just two lines.

conda activate dc4k
jupyter lab