A lakehouse gives a small data team one place for raw files, curated tables, and business metrics, with one governance model across all of them. The hard part is rarely the platform. It is deciding how data moves through the platform, who owns each layer, and how the team keeps the pipelines running after the consultants leave. Buoyant Data builds lakehouses on Databricks for mid-market teams that have a small in-house data group and need the result to be maintainable by that group.
Medallion architecture
The medallion pattern organizes tables into three layers, each with a clear quality bar. Databricks describes the layers as bronze (raw), silver (validated), and gold (enriched).
- Bronze holds data as it arrives from source systems, in its original form, appended over time. Keeping it lets you rebuild every later layer without going back to the source.
- Silver holds cleaned, validated, and deduplicated records, joined where sources need to be combined. This is where schema enforcement, null handling, and late-data rules live.
- Gold holds the tables people actually query: dimensional models, aggregates, and metrics shaped around business questions.
The pattern is a recommendation, not a requirement. For a small team, the practical value is that every table has a stated purpose and a stated quality level, so an analyst knows which layer to trust and an engineer knows where a fix belongs.
Unity Catalog for governance
Unity Catalog
is the governance layer for Databricks. It sits beneath queries and model
calls, enforces access control, records lineage, and writes an audit log. Tables
are addressed as catalog.schema.table, which maps cleanly onto the layers: one
catalog per environment or domain, and schemas for bronze, silver, and gold.
We set up Unity Catalog so that:
- Access is granted to groups, not individual users, so onboarding and offboarding do not require edits to every table.
- Each layer has its own schema with its own grants. Analysts read gold. Engineers write bronze and silver.
- Row filters and column masks cover sensitive fields where the business needs them, rather than maintaining copies of tables.
- Lineage is visible from source to dashboard, so a question about where a number came from has an answer in the catalog.
Unity Catalog is available for Databricks workspaces on AWS. Its privileges and object model are the same across clouds, so the design carries over if your platform later moves.
Lakeflow pipelines
Lakeflow pipelines are Databricks' framework for building batch and streaming transformations in SQL and Python. They are built on Apache Spark Declarative Pipelines, and Databricks describes the transformation code as portable to other SDP runtimes. You declare the tables and their sources, and the framework works out the dependency order and runs the updates.
For a small team this matters in three ways:
- The pipeline definition is the documentation. Reading the code tells you what each table is, where it comes from, and what it depends on.
- Ingestion, cleaning, and aggregation use the same model, so one engineer can own a flow from landing zone to gold table.
- Data quality rules sit next to the transformations that they guard. Bad records can be quarantined or the run can fail, and the event log records what happened.
We keep the pipeline count small and the naming consistent, so the team can see the whole flow without a diagram.
Migrating a legacy warehouse
Most mid-market teams are not starting from nothing. They have a warehouse, a set of ETL jobs, and reports that the business depends on. A migration to a lakehouse is a sequence of smaller moves, not a single cutover:
- Inventory. List the source systems, the tables and reports that consume them, and the jobs that move data between them. Identify the reports that matter most to the business.
- Map to layers. Decide what lands in bronze unchanged, what the silver model looks like, and which gold tables replace the existing marts. The Databricks data warehousing guidance describes how a warehouse's modeling work maps onto these layers.
- Run in parallel. Build the first gold tables, then compare their output with the warehouse for the same period before any report is switched over.
- Switch and retire. Move reports one group at a time, and turn off legacy jobs only after the comparison holds.
- Hand off. Document the pipelines, the grants, and the runbook, and make sure the in-house team can make the next change without us.
The comparison step is where most of the risk sits, so we plan time for it rather than treating it as a formality.
How we work
As a small consultancy, we aim to get your team building on the platform quickly rather than stay on as an ongoing dependency. A typical engagement runs in phases:
- Design: the layer model, catalog and schema layout, grant groups, and the first set of source systems.
- First pipeline: one source taken from landing zone to a gold table, with quality checks and documentation, to prove the pattern.
- Expansion: the remaining sources and reports, built by your team with us reviewing the work.
- Handoff: the runbook, the ownership map, and a short list of what to improve next.
We will tell you if a lakehouse is more than your team needs. A well-kept warehouse may be the right answer for now.
Planning a lakehouse build?
Contact us