Databricks bills by usage, so cost problems usually come from three places: compute that runs longer than it needs to, clusters and warehouses sized larger than their workloads, and spend nobody can attribute to a team or a project. Buoyant Data helps you find each of these in your account, then puts a small control plan in place so the same problems do not return.
Start with system tables
Databricks records billable usage in the system catalog, which you can query
with SQL like any other table.
system.billing.usageholds one row per billable usage record, with the SKU, the DBUs consumed, the workspace, and any custom tags. Records are typically available within 12 hours, and the table keeps 365 days of history. Usage is reported in DBUs, so it is the right table for trends and attribution. Billable usage system table referencesystem.compute.node_timelineholds minute-level utilization for classic compute, andsystem.compute.clustersholds each cluster's configuration over time, including its auto-termination setting. Together they show which clusters sit mostly idle and which are over-provisioned. Compute system tables reference
Tags and budgets
Cost attribution depends on tags. Databricks records custom tags with each usage
record, so budgets and dashboards can group spend by team, project, or
environment.
- Agree on a short list of tag keys before tagging anything. Tag keys and values
may contain only letters, numbers, and the characters
+ - = . , _ : @, and a workspace resource can carry up to 20 tags. Tag compute and workloads for cost attribution - Do not use the
Nametag key on a cluster. Databricks sets that tag itself, and overwriting it can stop the cluster from terminating when idle. Tag compute and workloads for cost attribution - For classic compute, a compute policy can require a tag with an allowed list of values, so clusters cannot launch untagged. Tag compute and workloads for cost attribution
- Serverless usage is tagged differently. A serverless usage policy applies its tags to the serverless notebooks, jobs, pipelines, and endpoints of the users it is assigned to. It does not tag classic compute, and it does not apply to existing assets until they are updated. Attribute usage with serverless usage policies
- Budgets in the account console track spend against monthly thresholds, scoped to the whole account or to selected workspaces and tags. A budget can have up to four thresholds, and it sends email when a threshold is crossed. Budgets are measured at SKU list price, so they do not reflect credits or negotiated discounts. Create and monitor budgets
- Budget alerts are not a spend cap. Email can arrive up to 24 hours after the
usage occurs, and
system.billing.usageremains the source of truth for what was billed. Create budgets to monitor spend
Cluster and warehouse right-sizing
Idle compute is the most common waste we find. Idle compute keeps accumulating DBU and cloud instance charges until it terminates.
- Set auto-termination on every all-purpose cluster. The inactivity period can be set from 10 to 10,000 minutes, and setting it to 0 turns it off. A compute policy can enforce a sensible default, such as one hour. Manage classic compute
- Run production jobs on job compute or serverless compute, so the compute ends when the work does. Best practices for cost optimization
- Use autoscaling with a realistic minimum and maximum worker count. Databricks documents that autoscaling can reduce overall cost compared with a statically sized cluster, although it has limits for scaling down structured streaming workloads. Best practices for cost optimization
- For SQL warehouses, prefer serverless where it is available. Serverless warehouses default to a 10-minute auto-stop and scale clusters up and down with demand. Pro and classic warehouses default to 45 minutes and scale on a slower schedule, so their auto-stop and cluster limits deserve a deliberate choice. Create a SQL warehouse
- Right-size a warehouse by starting larger and sizing down. Use the Peak Queued Queries metric on the monitoring page to decide when more clusters are needed, and check the query profile for bytes spilled to disk, which means the cluster size is too small. SQL warehouse sizing, scaling, and queuing behavior
A control plan
Savings persist only when the controls stay in place. The plan we propose has four parts.
- Tagging standard: the agreed keys, the enforced compute policy rules, and a serverless usage policy for each team.
- Budgets: one account-wide budget with two thresholds, plus a budget for each workspace or team that has its own spend owner.
- Monthly review: the baseline queries above, rerun each month, with the top three cost movers reviewed by the owning team.
- Guardrails: auto-termination enforced by policy, and job compute as the default for scheduled work.
We do not publish a savings percentage for this work. The right number depends on your workload mix, and we would rather measure it in your account than quote one from a general example.
Ready to find out where your Databricks spend is going?
Start with a baseline. We will show you where the money goes before we recommend any change.
Ready to cut your Databricks costs?
Contact us