Home › Guides › What Is Apache Spark
Tech Explained · 2026What Is Apache Spark? How It Works, Spark vs Hadoop and Your First PySpark Job in 2026
Apache Spark is an open source engine that splits a large dataset into partitions and processes them in parallel across many machines. A driver program plans the work, executors run it, and results stay in executor memory between steps instead of going back to disk. The current release is 4.2.0, published in July 2026.
- Spark is a scheduler, not a database. It reads files you already keep in object storage, cuts them into partitions, and hands the work to executors.
- The driver plans, the executors do. Nearly every confusing Spark error is really a question about which of those two ran your code.
-
200 is the number to remember. The Performance Tuning docs give
spark.sql.shuffle.partitionsa default of 200, and that one default explains a lot of slow jobs. -
No cluster needed to learn it.
pip install pysparkplusmaster("local[4]")gives you a real engine on a laptop. - Under about 50 GB, Spark is usually the wrong tool. DuckDB or Polars on one machine will finish before Spark has finished planning.
Your nightly job has been running for forty minutes. The Spark UI shows 199 tasks finished and one still going, and it has been that way for half an hour. Someone suggests more workers, you add four, and the job takes exactly as long. That is usually the moment people notice they have been using Spark without knowing what Apache Spark is doing with their data, and it is a shorter gap to close than the tutorials suggest.
How Apache Spark Works: Driver, Executors and Partitions
Spark has three moving parts, and once you can name them the error messages start making sense.
The driver is the process running your Python file, and it never touches your data. It hands your DataFrame code to the Catalyst optimiser, which rewrites it into a physical plan, then cuts that plan into stages and tasks. A planner with a clipboard.
The executors are separate JVM processes, usually on other machines, that read bytes and compute. Each gets a few cores, and each core runs one task at a time. A task always means "apply this piece of the plan to one partition".
A partition is a slice of your data. Read a 40 GB Parquet dataset made of 320 files and you get roughly 320 partitions, and Spark works on as many at once as you have cores. There is no magic in the Spark architecture, only the arithmetic of partitions divided by cores.
Apache Spark architecture: who talks to whom
Your code never touches data. It builds a plan that executors carry out one partition at a time.
Shuffle partition default checked against the Apache Spark Performance Tuning documentation, 4 October 2026.
Narrow and wide operations
A narrow operation such as filter or withColumn works inside one partition, so it costs almost nothing. A wide operation such as groupBy, join or distinct needs rows from every partition to meet, which means writing intermediate files and pulling them across the network. That is a shuffle, and it is the only part of Spark you have to think about actively.
Spark 4.2 facts worth knowing
Four numbers you will type or trip over in week one.
Spark 4.1.0 and 4.2.0 release notes and the Spark Connect overview, 4 October 2026.
How alive is the project
A fair question, given how often Spark is written off.
Apache Spark 4.2.0 release notes and Microsoft Learn Fabric Spark compute docs, 4 October 2026.
Apache Spark vs Hadoop: The Comparison People Actually Mean
Ask about Spark vs Hadoop and the real question is Spark against Hadoop MapReduce, the older batch engine. They stopped being rivals years ago. But why MapReduce lost tells you what Spark is for.
MapReduce wrote the result of every step back to disk, in triplicate, before the next step could start. A five step pipeline meant five round trips through HDFS. Spark keeps intermediate results in executor memory and spills to disk only when it runs out of room, so the same pipeline is one pass with four in-memory handoffs. That is the entire difference, and it is enough to turn hours into minutes.
| Dimension | Apache Spark 4.2 | Hadoop MapReduce |
|---|---|---|
| Intermediate data | Executor memory, spilling only when needed | Written to HDFS between every stage |
| What you write | DataFrames or plain SQL in Python, SQL, Scala, Java or R | Mapper and reducer classes, usually Java |
| Streaming | Structured Streaming on the same API as batch | Batch only |
| Optimiser | Catalyst, with Adaptive Query Execution on by default since 3.2.0 | None. Your code is the plan |
| Where you meet it in 2026 | Default engine in Databricks, Fabric, EMR and Dataproc | Legacy jobs nobody has rewritten yet |
Which do you learn? Spark, without hesitating. The one honest reason to understand MapReduce in 2026 is that some interviewers still ask, so spend twenty minutes on the concept rather than twenty hours. The closer modern comparison is Spark against single-machine engines: if your largest table is 20 GB of Parquet, DuckDB on a laptop beats a Spark cluster, because Spark pays its planning and startup cost whether you need the parallelism or not. Also read: What Is ETL, and When ELT Wins.
The Shuffle Is Where Spark Jobs Go to Die
Back to that stuck job with 199 tasks done and one running. Nothing is broken. One shuffle partition got far more rows than the other 199, because the key you grouped by is lopsided. Take a two-person data team at a mid-size Indian insurer grouping nightly claims by branch_code: their intake system writes the literal string UNKNOWN whenever the field is blank, and forty per cent of rows carry it. One task does forty per cent of the work, alone, while 199 cores idle. That is data skew, and adding workers cannot fix it, because you cannot parallelise a single partition.
The one number behind most slow Spark jobs
Every groupBy and join gets reshaped into this many pieces unless you say otherwise.
The documented default for spark.sql.shuffle.partitions, unchanged since Spark 1.1.0. On a 100 MB file that means 200 nearly empty tasks. On a skewed 2 TB join it means one task holding most of the data. Neither is what you want, and neither is Spark's fault.
Apache Spark Performance Tuning documentation, checked 4 October 2026.
What usually goes wrong here
When a job hangs, your instinct will be to open the cluster settings. Don't. Open the Spark UI, go to the Stages tab, click the running stage, and read the task duration summary: minimum, median, 75th percentile, maximum. Maximum twenty times the median means skew, and no amount of memory will help. Thousands of fast tasks means too many partitions and a scheduler bottleneck. Those two readings cover most of what goes wrong and take ten seconds.
Adaptive Query Execution helps. The Performance Tuning documentation describes it re-optimising the plan at shuffle boundaries from real runtime statistics, coalescing partitions that turned out small and splitting ones that turned out skewed, on by default since 3.2.0. It will not solve a single hot key. For that you salt: add a random suffix to the hot key, aggregate, strip the suffix, aggregate again.
Case study: 60 TB at Meta
The clearest public account of shuffle at the limit is still Meta's engineering write-up on engineering.fb.com of moving one of its largest Hive pipelines to Spark. The job read roughly 60 TB of compressed input and performed a 90 TB shuffle and sort, which the team called the largest real-world Spark job attempted by shuffle volume. The instructive part is not the size. Most of their work was not hardware; it was changing how shuffle files got written, fetched and tracked, because past roughly 10 TB of shuffle per job that traffic dominates wall clock time whatever else you tune. The lesson scales down to a 60 GB job: measure your shuffle first.
-
Default 200 on small data. Set
spark.sql.shuffle.partitionsto roughly two or three times your total core count. -
A hot key. Find it with
df.groupBy("key").count().orderBy(F.desc("count")).show(10), then salt it or handle it separately. - Joining big to small the slow way. If one side fits in memory, broadcast it. Spark often does this for you, but only when table statistics exist.
-
Calling
collect()to look at results. That pulls every row into the driver. Useshow(20)or write to a file.
PySpark Tutorial for Beginners: Your First Spark Job in Six Lines
No cluster, no cloud account, no Docker. Local mode runs the real driver and real executors as threads in one process, so everything you learn transfers.
pip install pyspark
python -c "import pyspark; print(pyspark.__version__)"Make a CSV called claims.csv with columns claim_id,branch_code,amount, or point this at any CSV you already have. Save the following as nightly_claims.py:
from pyspark.sql import SparkSession
from pyspark.sql import functions as F
spark = (SparkSession.builder
.appName("claims-nightly")
.master("local[4]")
.config("spark.sql.shuffle.partitions", "8")
.getOrCreate())
claims = spark.read.option("header", True).csv("claims.csv")
daily = (claims
.withColumn("amount", F.col("amount").cast("double"))
.groupBy("branch_code")
.agg(F.sum("amount").alias("total"),
F.count("*").alias("n_claims"))
.orderBy(F.desc("total")))
daily.show(10, truncate=False)
daily.explain("formatted")Run it with python nightly_claims.py, or through the submit script if you want the command you will actually use in production:
spark-submit --master "local[4]" \
--conf spark.sql.shuffle.partitions=8 \
nightly_claims.pyThree things in that file are worth breaking on purpose. Change local[4] to local[1] and the job takes about four times as long, which is the clearest proof you will get that Spark's speed is partitions divided by cores. Delete the spark.sql.shuffle.partitions line and watch the aggregation spawn 200 tasks for a few hundred rows. Then read what explain("formatted") printed, bottom up: scan, exchange, aggregate. Every Exchange is a shuffle, and counting them is the fastest performance review you can run on your own code. Also read: SQL Query Optimization in 2026.
The errors you will hit, and what they mean
| Symptom | What is actually happening | What to do |
|---|---|---|
Py4JJavaError and a wall of Java stack trace |
An executor failed. The Python error is only the messenger | Scroll to the first Caused by line. That is your real error |
| One task out of 200 never finishes | Data skew on a single key | Count your keys, then salt or isolate the hot one |
Container killed by YARN for exceeding memory limits |
Executor heap plus overhead passed its allocation, usually during a wide shuffle | Raise spark.executor.memoryOverhead, or add shuffle partitions so each is smaller |
| Fast locally, crawls in production | Thousands of tiny input files, so thousands of tiny tasks | Compact to files of roughly 128 MB to 1 GB first |
Where You Will Meet Apache Spark in Real Work
You rarely install Spark yourself any more. You meet it inside a platform, and the platform decides what you need to know.
In a Microsoft Fabric notebook. Anywhere in the Microsoft data stack, this is the Spark you will touch. Microsoft Learn documents the Fabric starter pool as giving a session in roughly 5 to 10 seconds because the cluster is already warm, and notes that custom libraries or Spark properties stretch startup to between 30 seconds and 5 minutes while the session is personalised. Starter pools run Medium nodes only and autoscale cannot be switched off. Writing PySpark against Lakehouse tables is graded territory on DP-700, and 360DT's Microsoft Fabric data engineering course practises those notebook and pipeline patterns live.
On a cloud cluster you own. EMR or Glue on AWS, Synapse pools or Databricks on Azure. The hard part stops being PySpark and becomes sizing, spot instances, IAM and VPC networking. A live AWS Solutions Architect and DevOps program or its Azure AZ-305 and AZ-400 equivalent is what makes Spark jobs cheap rather than merely correct.
Under a feature pipeline. Spark ML on Spark Connect reached general availability for the Python client in 4.1, and Spark is still the common way to build training features over history, next door to the packaging, drift monitoring and retraining an MLOps engineer program covers.
As a vector engine, newly. The genuinely new thing in 4.2.0: SQL primitives for vector similarity search, including distance and normalisation functions and NEAREST BY, a top-K ranking join for distance-based matching, plus native GEOMETRY and GEOGRAPHY types. Batch embedding search over a billion rows becomes a Spark SQL query instead of a separate database, which matters if you build retrieval systems in an AI engineer and RAG course. Also read: pgvector vs Pinecone.
Analysts meet Spark from the other side. A Power BI developer reading a Lakehouse is reading tables some Spark job wrote last night, and knowing why that job left 40,000 tiny files is the difference between a report that refreshes in a minute and one that times out. For that seat the Excel, SQL, Python and Power BI analyst track covers the consuming side, and the full certifications overview compares the two paths.
The Honest Caveat: Spark Is Oversold for Most Indian Data Teams
Most datasets at most companies are small. That insurer's entire claims history is probably under 200 GB, and the nightly increment is a few hundred megabytes. For that, Spark is overhead wearing the costume of rigour: cluster cost, startup latency, a JVM you debug through a Python wrapper, and a shuffle model to learn, in exchange for parallelism you do not need. Below roughly 50 GB, DuckDB or Polars on one machine is faster and easier to reason about. I would not learn Spark first today: SQL properly, then pandas or Polars, then Spark when a job stops fitting on one machine.
The counter-argument is employment, and it is fair. Spark is the default engine inside Databricks, Fabric, EMR and Dataproc, so job descriptions ask for it whether the company's data justifies it or not. Learn it for the interview if you must. Just do not let a tool choice pretend to be an architecture decision.
Learn Spark where you will actually use it: inside Microsoft Fabric
A live weekend program over 8 weeks covering Microsoft Fabric data engineering for the DP-700 certification, with DP-900 fundamentals included. Hands on projects, mentor support and placement guidance, so you write PySpark against real Lakehouse tables instead of reading about partitions.
Explore the course
How to Learn Apache Spark Properly
Three weeks of focused evenings is enough, in the right order. Week one, local mode only: read a CSV, run a groupBy, read the physical plan, then break it by removing the shuffle partition config so you watch those 200 tasks appear. Week two, one platform: a Fabric notebook, writing to a real table and reading it back from a BI tool. Week three, the Spark UI: take a working job, make it slow on purpose with a skewed key, fix it with salting and a broadcast hint.
Skip RDDs. Older tutorials open with them and almost no production code uses them now, because DataFrames get Catalyst optimisation and RDDs do not. Skip cluster installation too; nobody will ask you to put Spark on bare metal.
In your position I would not buy anything yet. Spend one evening on the six lines above, and if the Spark UI turns out interesting rather than tedious, sit in on one of 360DT's free webinars, where the Fabric and Spark material is demonstrated end to end. That tells you whether data engineering suits you far more cheaply than a course does.
Related guides
- What Is Apache Kafka? the streaming half of the same architecture, where partition means something different.
- Microsoft Fabric vs Databricks in 2026 both run Spark underneath, so this is the decision you will actually face at work.
- Apache Airflow Tutorial 2026 something has to trigger your Spark job every night, and this is usually it.
- Data Engineer Roadmap 2026 where Spark sits in the full sequence of skills, with realistic timings.
- dbt Tutorial for Beginners 2026 the SQL-first alternative to writing transformations in PySpark at all.
- Data Engineering Projects for 2026 six builds to drop a Spark job inside.
Frequently asked questions
What is Apache Spark used for?
Four things: batch transformation of large files into clean tables, streaming through Structured Streaming, feature engineering for machine learning, and ad hoc SQL over data too large for one machine. Since 4.2.0 it also does vector similarity search and geospatial queries natively.
Is Apache Spark still relevant in 2026?
Yes, for a less exciting reason than it used to be: it is the default engine inside Databricks, Microsoft Fabric, AWS EMR and Google Dataproc, so most managed data platforms are Spark with a different front door. The 4.2.0 release closed over 1,700 Jira tickets with more than 250 contributors.
Do I need to know Java or Scala to learn Spark?
No. PySpark and Spark SQL cover nearly everything a data engineer does, and 4.2.0 enabled Arrow-optimised Python UDFs by default, removing much of the old penalty for staying in Python. You will still read Java stack traces when executors fail, but reading them is not writing Java.
What is the difference between Spark and PySpark?
Spark is the engine, PySpark is the Python API you drive it with. Calling a DataFrame method builds a plan that the JVM executes later, which is why a Python typo surfaces instantly but a bad cast only explodes once an executor runs.
How much data do you need before Spark is worth it?
When one job no longer fits comfortably on one large machine, which in practice is somewhere above 50 GB of compressed Parquet, or when processing overruns your nightly window. Below that, DuckDB or Polars is usually faster and much cheaper to operate.
What is a shuffle in Spark and why is it slow?
A shuffle is the step where Spark regroups rows by key so a groupBy or join can work. It writes intermediate files on every executor and fetches them across the network, far slower than computing inside a partition. It is also where skew appears, because one key can land most of the rows in a single partition.
Can I run Apache Spark on my laptop?
Yes, and you should start there. pip install pyspark then master("local[4]") runs the real driver and real executors as threads in one process. Partitions, plans, shuffles and the Spark UI all behave as they will on a cluster, so local mode is the cheapest place to learn the parts that matter.
About this guide. 360 Digital Transformation is an Anthropic Partner. Other certification bodies, vendors and employers named here are not affiliated with us. Tools and versions change quickly; commands and figures cited were checked on 4 October 2026.
Microsoft Fabric
Python
