kacem.dev
← DS · Distributed systems

Spark 4 on a modern JVM: Java 21, Java 25, and the APIs worth caring aboutDraft

What changed in Apache Spark 4, how it behaves on Java 21 and the Java 25 LTS, and the API additions that actually alter how a risk engine gets written.

4 min sparkjavajvmdatascala

Spark 4 is the first major release in years that changes the ground a cluster stands on rather than just the roof. Scala 2.12 is gone, the Java floor moved, ANSI SQL semantics are the default, and Spark Connect is no longer an experiment. This post is my notes from moving the VaR platform (market-risk-quant) onto it, plus a running log of releases as they land.

I run Spark 4.0.0 on Scala 2.13 and Java 21 in production today. The Java 25 section below is what I have tested and what I have not.

The news, compressed

  • Scala 2.13 only. 2.12 artifacts are gone. If you have a _2.12 suffix anywhere in your build, Spark 4 is a migration, not an upgrade.
  • Java floor raised. Spark 4.0 targets Java 17 and 21; Java 8 and 11 are out.
  • ANSI mode on by default. Integer overflow, bad casts and out-of-range dates now throw instead of silently returning null. For risk numbers this is the single most important change in the release and the one most likely to break a pipeline that “worked” for years.
  • Spark Connect is the client story. A thin client talks gRPC to a long-lived server; the driver no longer lives in your process.
  • New types. VARIANT for semi-structured data, and better collation support in SQL.
  • SQL pipe syntax and session variables. Small on paper; large for the kind of ad-hoc analysis that precedes every new risk measure.

Java 21: what the upgrade bought

Moving from Java 17 to 21 under Spark 4.0 was uneventful, which is the best thing a JVM upgrade can be. The things that measurably mattered:

Generational ZGC. Executor pauses that used to show up as straggler tasks on the wide shuffle before the P&L aggregation disappeared. Flags I settled on:

-XX:+UseZGC -XX:+ZGenerational -XX:SoftMaxHeapSize=...

Virtual threads, carefully. They are a fit for the driver side — the Spring Boot 4 service that fans out scenario requests — and not for executor task code, which is already parallel by construction. Pinning on synchronized blocks inside older JDBC drivers is the usual trap.

Records and pattern matching make the Java-side DTOs between the Spring service and the Spark job much smaller. Not performance, just fewer lines to get wrong.

Java 25: tested and untested

Java 25 is the current LTS. What I have actually done:

  • Built and ran the full VaR batch on a Java 25 runtime against Spark 4.0.0 with --add-opens flags copied from the 21 config.
  • Checked that nothing in our code touches APIs removed or deprecated-for-removal in 25 (the SecurityManager removal is the one that bites libraries, not applications).

What I have not done is run it in production. Until Spark lists the JDK as supported, a green test suite is a data point, not a decision.

API enhancements that change the code

Most release notes are additive sugar. These are the ones that altered how I write a job.

1. ANSI by default forces explicit intent. Every cast that could overflow is now a design decision: try_cast, try_add, try_divide where I want null, the plain operator where I want the job to fail. A VaR number that is silently null for one desk is far worse than a job that stops.

2. VARIANT for the trade blotter. Upstream trades arrive as JSON with a schema that drifts per asset class. Landing them as VARIANT and projecting with variant_get removed a brittle “schema registry” we maintained by hand.

3. Spark Connect decouples the service from the cluster. The Spring Boot API used to embed a driver, which meant redeploying the service redeployed the Spark session. With Connect the service is a stateless gRPC client; cluster upgrades and service deploys are independent events.

4. Python data source API. Not for the batch itself, but for research: a small custom source that reads our scenario store directly into a DataFrame, so the notebook and the job see the same rows.

5. Structured Streaming state store improvements and transformWithState. Relevant for intraday risk, which is next year’s problem. Noted here so I remember why I cared.

What I ignore

  • Benchmarks that compare Spark 4 to Spark 3 on TPC-DS. My workload is nothing like TPC-DS.
  • Connectors I don’t use.
  • Anything about the Kubernetes operator until it is the default deployment path.

Release log

  • 4.0.0 — the baseline for everything above. Scala 2.13, Java 21, ANSI default. In production for the VaR batch.
  • 4.1.x —
  • Java 25 officially supported in —