Borrow a cluster in the cloud

This page walks the grown-up through the sign-up — whether you're setting this up for a kid or working through the book yourself. Budget about ten to twenty minutes, most of it waiting for confirmation emails.

Which one should we pick? 🤔

Holden has worked at or with Google Cloud, Amazon AWS, Microsoft Azure, IBM Cloud, Snowflake, and Databricks — enough of them that there's no favourite left to have. All of them will run the code from this book, and all of them will tell you at length that their version is the fast one. The buttons live in different places and everybody uses slightly different words for the same things, but underneath it's the same Spark.

Two of those words are worth learning before you choose, because between them they decide almost everything about cost. A cluster is a pile of computers you rent by the second, and it keeps billing whether you're using it or not. Serverless means the provider switches computers on when you run something and stops charging when you stop. Wherever we get the choice below, we pick serverless.

🧱 Databricks

Databricks was started by the people who created Spark, and their free tier is the gentlest way to get a Spark notebook. No credit card, and no cluster to remember to turn off.

1. Go to databricks.com/learn/free-edition and sign up for Databricks Free Edition. (Older books and blog posts talk about Community Edition. That was the previous free tier; it was retired in 2025 and Free Edition replaced it. They're not the same product and old logins don't work, so follow these steps rather than the old ones.)
2. Confirm your email address and log in.
3. In the left sidebar click New → Notebook.
4. Make sure the language selector at the top of the notebook says Python.
5. Type spark.range(1000000).filter("id % 7 == 0").count() into the first cell — a cell is one of the boxes you type code into — and press Shift+Enter, holding Shift and pressing Enter, which is how you tell a notebook "run this now."

The compute starts itself the first time you run a cell — that can take a minute — and then you should see 142858.

Free Edition is capped in size and idles itself out when you stop using it, which is exactly what we want — there's no payment method attached, so it cannot bill you. Two things to expect: if you use up the daily allowance your compute stops until tomorrow, and when a notebook idles out it forgets its variables, so you re-run your cells. If you outgrow it, Databricks also offers a full trial that runs inside an AWS, Azure, or Google Cloud account — that one can cost money, so read the tidying up section first.

🔷 Google Cloud

Google Cloud's Spark service is called Managed Service for Apache Spark. You'll also see it called Dataproc — that's the older name, and it's still the word used in the command-line tool and in permission names, so don't be thrown when the branding and the commands disagree. New accounts get a free trial credit, and a card is required to prove you're a human. Google says it won't charge you when the trial ends unless you choose to upgrade.

1. Sign up at cloud.google.com/free with a Google account.
2. Create a project — think of it as a labelled box that holds everything you make. Call it something like dc4k.
3. Set a budget alert: search the console — Google's name for its control-panel website — for Billing → Budgets & alerts, and make a budget for a small amount like $5 so you get an email if something you forgot about is still running.
4. Search the console's top search bar for Dataproc and enable the API when it asks.
5. The friendliest option is the serverless flavour: choose to create an interactive session, or open BigQuery Studio and start a notebook with a Spark session attached. Serverless means there's no cluster sitting around costing money between uses.
6. If you'd rather have a classic cluster, use Create cluster, pick a small 2-vCPU machine type such as e2-standard-2 (the very smallest "shared core" types like e2-micro aren't allowed here, so don't waste time trying them), tick Jupyter under optional components, and — importantly — set scheduled deletion so the cluster switches itself off after an hour or two of idleness.

!apt-get -qq install -y openjdk-17-jdk-headless
!pip install -q "pyspark[sql]==4.2.*"
import os
os.environ["JAVA_HOME"] = "/usr/lib/jvm/java-17-openjdk-amd64"

📦 Amazon AWS

Amazon's Spark service is EMR — Elastic MapReduce, named after Hadoop MapReduce, the older idea Spark was invented to improve on. AWS has the most steps of any provider here. EMR has no permanently-free tier of its own, but new AWS accounts come with sign-up credits that EMR usage can draw on. Keep an eye on the meter either way.

1. Sign up at aws.amazon.com/free. You'll need a card and a phone number.
2. Turn on a budget straight away: search for Billing and Cost Management → Budgets, and create a small monthly budget with an email alert.
3. Pick a region in the top-right corner — somewhere near you — and stick with it. Things you make in one region are invisible in the others, which confuses absolutely everybody at least once.
4. Search for EMR, then choose EMR Serverless and create an application with the Spark option. Two settings matter before you click Create: tick the box that enables interactive use (it's off by default, and without it the notebook step below has nothing to attach to), and leave the "pre-initialized capacity" setting alone — that option keeps machines warm and ready, which also means it keeps them billing. Serverless is the right choice for us: it bills for the time your work is actually running rather than for a cluster sitting idle.
5. For a notebook, open EMR Studio and create a Workspace, then attach it to your serverless application. Alternatively, AWS Glue interactive sessions give you real PySpark with less clicking.
6. Some steps ask you to create an IAM role or an S3 bucket. Accept the defaults it offers — those are the permission slip and the storage locker, and AWS will make sensible ones for you.

🟦 Microsoft Azure

Azure has more than one Spark: Microsoft Fabric (the current recommendation, with Spark notebooks built in), Azure Synapse Analytics Spark pools, and Azure Databricks, which is Databricks living inside Azure.

1. The quickest path is Microsoft Fabric: sign in with a Microsoft account and start the Fabric trial. The trial doesn't need a credit card. It usually does need a work or school email address, not a personal Outlook or Hotmail one. So: if you have a work or school email, this is the fastest card-free path on the page. If you don't, it's worth asking a parent or helper whether they can sign up through their workplace — and if nobody has one, use Databricks Free Edition instead.
2. In Fabric, create a workspace, then inside it create a Notebook. Fabric notebooks are PySpark notebooks with spark already defined, so run spark.range(1000000).filter("id % 7 == 0").count() and you're off.
3. Good news on safety: Azure free accounts come with a real spending limit switched on by default, equal to your trial credit. Leave it on. Under Cost Management + Billing you can check it's on and add a budget email alert — but note that budgets only send warnings, while the spending limit is the thing that actually stops the meter. You can't set a custom amount; it's fixed at the credit.
4. Want Azure Databricks instead? Be aware this is a bigger commitment than the Fabric trial: its free trial won't run on a free-trial subscription, so you have to convert to pay-as-you-go and give up that spending limit first. If you're going to do that, set a budget alert before you create anything, and read the tidying up section. For learning the book, Databricks Free Edition gets you the same notebook with none of this.

🔵 IBM Cloud

IBM's notebook environment is part of watsonx.ai Studio — you'll see it called Watson Studio in anything written before late 2024. IBM's free "Lite" plans genuinely never bill you, but IBM Cloud does ask for a credit card when you create the account, purely to check you're a real person.

1. Sign up at cloud.ibm.com/registration and verify your email.
2. From the catalog, add watsonx.ai Studio and choose the Lite plan. Lite plans are free and simply stop when you've used up the month's allowance — they don't bill you. The allowance is about ten hours of compute a month, which is plenty for working through chapters but not for leaving things running.
3. Create a project, which is IBM's box for holding notebooks and data. Some projects ask you to associate a Cloud Object Storage instance; pick the Lite plan for that too.
4. In the project, add a new asset of type Notebook. When it asks you to choose a runtime — that's the kind and size of computer your code will run on — pick one whose name includes Spark, for example "Default Spark 4.0 & Python 3.11". On Lite you get two executors, which is a small but genuine cluster.
5. IBM doesn't reliably hand you a ready-made spark variable, so start the notebook with the explicit version — it's correct everywhere:

   from pyspark.sql import SparkSession
   
   spark = SparkSession.builder.appName("dc4k").getOrCreate()
   spark.range(1000000).filter("id % 7 == 0").count()

Lite runtimes stop your notebook once it's been idle a while. Mildly annoying mid-exercise; excellent for the family bank account, since nothing can quietly run up a bill while you're at school.

❄️ Snowflake

Snowflake gives you a 30-day trial with a chunk of free credits and no credit card, and it has notebooks built in. One thing to know up front: Snowflake's own DataFrame library is called Snowpark. It was deliberately designed to look like Spark, so most of the code in this book reads the same — but it isn't literally Spark, so a few things are spelled differently.

1. Sign up at signup.snowflake.com. Any cloud provider and region will do. (If you let the trial lapse and want it back later, reactivating it does need a card — so use it while it's fresh.)
2. Log into Snowsight, the web interface, and go to Projects → Workspaces, then create a notebook there. Plenty of guides still say Projects → Notebooks; that was the older layout, and new accounts can't use it, so Workspaces is the one to look for.
3. Notebooks run on Snowflake's own compute, and only reach for a warehouse when you run SQL or Snowpark — which our example does, so pick the smallest one offered (X-Small). A Snowflake warehouse is its word for the cluster doing the work: confusingly, it means the computers, not the data they're working on, which trips up nearly everyone who has heard "data warehouse" used the other way. It auto-suspends when you stop using it, which keeps the trial credits from evaporating.
4. To use Snowpark, start your notebook with:

   from snowflake.snowpark.context import get_active_session
   
   session = get_active_session()
   session.range(1000000).filter("id % 7 = 0").count()

   You should get 142858, same as everywhere else. Note the single =: that's SQL's way of saying "equals", where Python wants ==.
5. If you'd rather run genuine PySpark code against Snowflake, look up Snowpark Connect for Apache Spark in their docs — it accepts real Spark DataFrame code and runs it on Snowflake's engine. It targets the Spark 3.5 API rather than the 4.2 we use, and it leaves out the older RDD API, streaming, and MLlib, so treat it as a very good translation rather than a perfect one.

Does it work? The same test everywhere 🧪

Whichever provider you picked, run this in the first cell of the notebook:

spark.range(1000000).filter("id % 7 == 0").count()

If the answer is 142858, you have a working Spark cluster and you're ready for the book. 🎉 (Why that number? There are 142,857 multiples of seven under a million — 142857 × 7 = 999999 — plus zero, which counts too.)

If it complains that spark is undefined, the notebook probably isn't a Spark notebook — or it's still starting up. Try adding this above it:

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("dc4k").getOrCreate()

If that fails too, the notebook is a plain Python notebook. Either pick a Spark runtime when you create the notebook, or use the Colab recipe above — installing PySpark and pointing it at a Java — which gives you a one-machine Spark, plenty for the exercises.

Tidying up — the important bit 🧹

Cloud computers keep charging while they're switched on, even if nobody is looking at them. Make this a habit at the end of every session:

• Databricks Free Edition, IBM Lite, Snowflake trial, Colab, Fabric trial: nothing to do. They stop themselves and can't bill you.
• Any cluster you created by hand — a Dataproc cluster, EMR cluster, Synapse pool, or Databricks trial cluster — needs to be deleted or terminated. Not "stop the notebook": the cluster itself.
• Check the bill the next day. Open the billing page and look at the current month's charges. If it says a few cents, or nothing, you're fine. For scale: doing the book's exercises on a serverless option costs pennies per session, a small cluster left switched on by mistake is a few dollars a day, and a big one can be a few hundred. That gap — between what you actually used and what you forgot to turn off — is where essentially every surprising cloud bill comes from. This thirty-second check is how you catch it before it matters.
• Serverless options (serverless Dataproc, EMR Serverless, Databricks serverless) only bill while something is actually running, which is exactly why we recommended them. There are, of course, servers. "Serverless" means somebody else has to remember to switch them off — which is the part we care about.
• Done for good? Close the account, or at least remove the payment method. Every provider on this page lets you do that from the billing page.

When things go wrong 🔧

The sign-up won't accept the credit card

Providers usually put a small temporary hold (about $1) on the card to check it's real, and some prepaid and virtual cards get rejected for this. Try a different card, or use one of the genuinely card-free options: Databricks Free Edition, Snowflake's trial, or the Fabric trial if you have a work or school email. (IBM's Lite plans never bill you, but IBM still wants a card at sign-up, so it's not a way around this one.)

"Quota exceeded" or "insufficient capacity" when starting a cluster

New accounts start with low limits, and a region can genuinely be full. Ask for a smaller cluster (fewer machines, smaller machine type), or switch to a different region. Both are normal, and neither is your fault.

The cluster takes ages to start

Five minutes is normal for a real cluster — the provider is finding computers for you, switching them on, and installing Spark on each one. Serverless options usually start in under a minute.

A permissions or "access denied" error

Cloud accounts are strict about who is allowed to do what, and the person who made the account has to grant permissions to everybody else — including, sometimes, to themselves. Sign in as the account owner and accept the default role the wizard offers.

The button in the instructions isn't there

Very likely the product was renamed or the page was redesigned — it happens constantly, and this page names three renames already. Search the provider's own docs for the bold words in the step rather than hunting the menus.

The notebook can't find your data file

Cloud clusters can't see the files on your laptop. Upload the file to the provider's storage first (S3, Cloud Storage, ADLS, Cloud Object Storage, or the notebook's own file browser), and use the path it gives you.

We got a bill! 😱

It happens, and it's usually a forgotten cluster. Delete anything still running, then contact the provider's support — all of these companies have a history of forgiving first-time accidental charges, especially for people who are learning.

This is all too much

Completely reasonable. Try the Colab recipe, which is the shortest path to a working notebook on this whole page, or install Spark on your own computer. The book works fine on one machine — a cluster just makes it more exciting.