The most frequently asked question I got in 2025 was: how do you design for high-throughput with Delta Lake? The design of Delta Lake supports extremely high-throughput writes but the strategies for getting that high-throughput are not entirely obvious. The two key concepts in Delta Lake are a transaction log which tracks the table's state and data files which contain the rows of the table. Each presents different scaling constraints which must be accounted for in different ways, depending on the application. In this post I'll review a couple of strategies and their real-world implications.
My previous post about the concurrent transaction race problem with Delta Lake is useful to read, but a quick overview:
- The monotonically increasing version number coupled with other potential state changes typically requires a reload of state between writes.
- Delta writers therefore typically scale throughput by scaling the sizes of a batch rather than the number of concurrent batches.
Delta writers should be scaled vertically rather than horizontally.
The "Buoyant Architecture" exhibits this simple principle in not entirely obvious ways. We typically associate vertical scaling with "we've got to get a bigger box." In the Buoyant Architecture vertical scaling means separating the data write from the transaction write to allow for massive increases in data volume.
The foundation of the Buoyant Architecture is oxbow, a simple service usually deployed in AWS Lambda.
Oxbow
The medallion architecture plays an important role in the Buoyant Architecture. Data ingestion should almost always be considered as part of the bronze layer, into which semi-structured append-only data is landed. Some organizations will consider their unstructured data as bronze but I would consider that raw data which should be handled differently, outside of the medallion architecture in a modern data lake.
Oxbow allows an applications to create Apache Parquet data files directly into object storage like Amazon S3. After a data file has been written S3 sends an event notification to either SNS or SQS which triggers Oxbow.
flowchart TD
Input["Producer"]-- receive data ---WriterA
Input --> WriterB
Input --> WriterC
WriterA-- write parquet ---Storage
WriterB --> Storage
WriterC --> Storage
Storage --> Notify["Event notifications"]
Notify-- Batched events ---Oxbow
Oxbow-- transaction ---Delta["Delta table"]
By separating data writes from transaction writes, the ingestion and writing of data can be horizontally scaled with the needs of the data producer. Serializing transactional appends to the Delta table through SQS ensures a fast and serialized commits to a transaction log. The performance of these appends does not change with the data size, meaning an Oxbow lambda which processes a stream of thousands of 1GB files is functionally equivalent to one which processes thousands of 100GB files.
In the AWS ecosystem alone there now dozens of ways to have cloud-native Apache Parquet data generation. AWS Aurora can export snapshots to Parquet in S3. AWS Data Firehose (formerly Kinesis Data Firehose) can produce Parquet into S3 using its output conversion system. Airbyte can produce Parquet into S3. Practically every data system can produce Parquet into S3.
When we haven't been able to use a cloud-native service to produce Parquet data, such as with high volumes of syslog data, we have instead opted to deploy services like hotdog which only ingests data into Apache Parquet. From there we can turn that into Delta tables with ease.
Oxbow makes it simple to turn any appendable data stream into bronze layer Delta tables.
sqs-ingest and file-loader
Not everything will produce an Apache Parquet file into S3, which is why the Oxbow suite also contains two other really useful utilities: sqs-ingest and file-loader. Unlike the original oxbow Lambda these utilities cannot separate data and transaction writes, which makes scaling them more difficult.
For smaller serverless workloads we use the sqs-ingest Lambda to allow
applications to send rows directly into AWS SQS as JSON serialized records.
sqs-ingest supports many different workloads which need a quick and easy way
to produce bronze-layer data, such as other Lambdas or other online
applications. Recently AWS increased message size
limits
to 1MiB for SQS which has allowed more applications to emit append-only data,
but the size limit can cause trouble for systems which require larger payloads.
Larger payloads use file-loader which effectively just acts as a dumb sink
for line-delimited JSON (JSONL) files written into S3. Similar to oxbow it
receives S3 event notifications and then reads the input files to produce a
transaction in a Delta table.
As both of these tools see increased throughput, their performance can suffer with lots of small commits to the Delta table. Even with regular checkpoint intervals transaction-bloat for an ingestion system under heavy load can make performance worse!
The key is to make each invocation of sqs-ingest or file-loader bring as
much data as possible into a given transaction. AWS Lambda has a 6MiB limit
on the amount of data that can be passed when a Lambda is triggered, which
presented a ceiling on the amount of data which could be written per-execution.
I discussed the challenges with my pal Jared Wiener at AWS
who suggested a simple and novel approach which has since been incorporated into both sqs-ingest and file-loader: "just go get more messages!"
Both tools now vertically scale with an optional setting of
BUFFER_MORE_MESSAGES. The Lambdas are configured to use SQS as a trigger but
only for one record. This means that the Lambdas won't execute if there is no data available, keeping costs down, but once triggered they will attempt to buffer as many messages as possible per invocation:
flowchart TD
Input["Producer"]-- write rows ---Queue["SQS"]
Queue-- trigger message (1) --> WriterA["sqs-ingest"]
WriterA-- fetch more --> Queue
WriterA-- fetch more --> Queue
WriterA-- commit transaction ---Delta["Delta table"]
In the case of sqs-ingest it will attempt to buffer messages up until
approximately 50% of the Lambda functions configured runtime. For file-loader
it is governed both by the runtime and the amount of memory configured to load
JSONL data from S3.
Refrain
In a nutshell the Buoyant Data architecture builds on the medallion architecture and involves separating the components of the data ingest pipeline. Each component can have different scalability and performance characteristics. With Delta Lake this means separating the writing of the data files from the transaction log along process boundaries whenever possible.
When that separation isn't possible the approach is to make each write of data as large as possible to reduce transaction log bloat.
how do you design for high-throughput with Delta Lake?
- Separate data from transaction writing into different processes when possible.
- Serialize transaction writes to Delta table
- Perform downstream transformations or joins for silver layer in an event-driven manner off the bronze table.
Adopting cloud-native and serverless technologies as part of this approach we have repeatedly deployed data platforms which are 1-2 orders of magnitude cheaper than their predecessors.
If your team is interested in building out serverless data ingestion pipelines drop me an email and we'll chat!!
