5 tips for cheaper Databricks workloads

May 21, 2023

by Lizzie M.

databricks

aws

Databricks bills track utilization, which means most overspending is self-inflicted: clusters configured once and never revisited. The five changes below are the first things we check in any cost review, because they are low-risk and pay for themselves within a billing cycle or two. None of them require re-architecting anything.

One note on framing before the list: it is tempting to attack the Databricks line item itself, but in our experience a well-run Databricks platform is cheaper than an equivalent platform hand-built on raw AWS, GCP, or Azure services. The goal is not to spend less on Databricks; it is to get more out of every dollar you already spend on it.

Use Photon

Photon is Databricks' optimized Spark runtime, and it shortens run times for most SQL, Python, and Scala workloads on Apache Spark. Shorter jobs mean fewer compute hours billed. The business case is simple: same work, less time, lower invoice.

Photon falls back to the standard engine for operations it does not support, so it is safe to enable by default. The caveat: DataFrame-based jobs see the speedup; RDD-based workloads and driver-bound jobs generally do not. It is a checkbox at cluster launch, which makes it the cheapest experiment on this list.

Spot-only workers with Fleet Clusters

Most clusters we audit run an On Demand driver plus Spot workers with "fall back to On Demand" enabled. For many workloads that fallback is a quiet budget leak: Spot-only workers are cheaper and safe for jobs that tolerate variable capacity.

Pair this with Fleet Clusters, which hunt for Spot capacity across availability zones and instance types, and the platform finds the cheapest compute available without your team managing instance lists.

When Spot capacity runs short, the job runs with fewer workers instead of failing. Our testing consistently shows that a slightly longer job on fewer Spot instances costs less than falling back to On Demand.

One caution: Spot competition varies by region and can move prices. For North American customers we recommend us-east-2 and us-west-2.

Reduce additional EBS

EBS is the sleeper line item: for many customers it approaches the size of their compute bill. By default Databricks provisions two volumes per worker (30GB for the OS, 150GB for the worker and logs). Costs balloon when teams attach additional volumes to paper over shuffle- heavy or otherwise poorly tuned jobs. In most cases the right fix is the query, not more disk.

The common counterargument is extra EBS for Delta caching. In our measurements the hit rates for most jobs are too low to pay off; for the majority of workloads it is spend without a corresponding performance gain.

Use SQL Serverless

With SQL Serverless, compute runs in Databricks' account rather than yours. The feature that matters financially is provisioning in seconds: warehouses can be configured with zero minimum capacity, which means they cost $0 while idle. Users get clusters in seconds when they start querying, and the warehouse shuts down rapidly when they stop.

For interactive SQL and BI workloads with spiky usage, this converts a standing cost into a pure pay-per-use one, and lets you benefit from the pricing a buyer of Databricks' scale commands from AWS.

Avoid certain instances

Do not use i3 instances on AWS. They were once the path to fast local NVMe for Photon, but they are among the most contested Spot families, which means frequent interruptions and forced On Demand fallback. If your workloads benefit from local storage, use m6d or c5d families instead, and consult the AWS Spot Instance Advisor for interruption rates in your region.

Better still: rely on Fleet Clusters instead of hard-coding instance types. It is the lowest-maintenance way to keep landing on well-priced capacity as AWS shifts the market under you.


These five changes reliably produce a visible drop in the next monthly bill. After the easy wins, further savings come from measurement and experimentation, which is where engagements tend to pay for themselves. If you want a cost review of your platform, send us a note.