oxbow 1.7.0 changelog

v1.7.0 — released 2025-10-09

Changes in this release (all commits).

• Prepare the parquet-concat lambda for release with 1.7.0
• Add parquet-concat lambda
• Turn off expired logs cleanup
• Fix list coercion
• Revert "Upgrade oxbow to deltalake 0.28"
• Utilize the same table schema reference through the duration of file-loader
• Hold onto the table schema to avoid RecordBatchWriter mutations
• Properly iterate through record batches on deserialization
• Upgrade oxbow to deltalake 0.28
• Wrap unit test calls to set_var in unsafe {}
• Produce a small completion marker file when cdf-to-csv completes
• Coerce timestamps inside structs and lists
• Ensure that file-loader batches are augmented when appropriate
• Datafusion is unfortunately still required for tables with invariants
• Temporarily disable arm64 builds which are too slow during release
• Introduce significant changes to the file-loader Lambda to handle larger inputs
• Bump for a minor release
• Rely more heavily on arrow json deserialization for file-loader
• Bump to 1.4.2 for cdf-to-csv deletes
• Support generating delete information for the cdf-to-csv function
• Propogate S3 errors up to fail the file-loader lambda
• Lowercase the column name when glue-sync runs
• Generate the S3 object store from the environment variable
• Fix a regression where records_with_url_decoded_keys was dropping _delta_log changes
• Ensure data deserialization failures bubble up an error
• Bump the minor version for the substantial cdf-to-csv changes
• Utilize Datafusion to produce the output CSV files from the change data
• Add a cdf test table from the delta-rs tree
• Introduce the TableTrigger abstraction for detecting changes better
• Prevent temporary dir from being dropped in the middle of the test
• Upgrade to deltalake 0.26 which has some minor API changes
• Coerce nanosecond timestamps into timestamps of some form for delta purposes
• Add some cleanups and clippy fixes
• CDF to CSV lambda
• Fix issue with oxbow's commit_to_table not utilizing the table checkpoint retention
• Update to the latest 0.25 for delta-rs
• Properly percolate write errors up
• Clean up visibility timeout and SQS settings for more aggressive SQS consumption
• Reduce potential for empty receives in the buffer more functionality of sqs-ingest
• Properly delete manually fetched messages from SQS
• Allow sqs-ingest to fetch more messages than what it is triggered with
• ci: ensure the release can build arm64 binaries
• Update the release action to be more modern and support arm64 builds
• Return the schema evolution field coercion with the newer delta kernel support
• Remove unnecessary escaping of SNS->SQS records
• Introduce UNWRAP_SNS_ENVELOPE to sqs-ingest
• Remove hard-coded checkpointing from the write path and rely on the delta-rs checkpoint
• Properly pin the latest delta-rs with a modified delta-kernel
• Avoid binstall in the release
• Ensure cross is installed in github actions for releases
• use the cross image for building the release
• bump patch version for a rebuild of release
• Pin to a delta-kernel-rs which can be built for Lambda
• Upgrade to the released deltalake 0.23 with concurrency improvements
• Upgrade to the pre-release of deltalake 0.23 which includs concurrency fixes
• Revert "update to the latest delta-rs which has a few performance improvements"
• update to the latest delta-rs which has a few performance improvements
• Release the file-loader lambda on GitHub
• upgrade version
• upgrade lambda-http
• Switch append_values() to accept an iterator rather than a as_slice
• bump to 1.0.8 and incorporate the latest released delta-rs
• Use urldecoded s3 key in auto-tag lambda
• Update to merged code in delta-rs
• bump to the fix checkpoint creation issue in 1.0.5
• Build a release with lower checkpoint memory pressure
• Fix conditional gate for checkpoint creation in oxbow
• Ensure that checkpoints are never attempted on the initial creation
• fix: correct duplicate detection on TableMods
• chore: mark 1.0.1 for a new build
• chore: upgrade to the latest delta-rs for some more bug fixes
• fix: ensure redundant files cannot be added/removed in the same transaction
• tag rc3
• Clean up the specific semantics around the DynamoDB logstore usage and locking
• adjust for change in signature of log_store_for() in deltalake
• upgrade to deltalake 0.19.0 which has been released with loads of improvements
• Ensure that ziglang is installed for CI
• Relocate the Jenkinsfile to get picked up by internal systems
• Upgrade to the latest deltalake 0.18.2 with some API changes
• Bump the patch version for release
• Fix mistake in glue-create lambda (incorect handling of create database if not exists)
• Remove the unnecessary table load in the oxbow checkpointing flow
• Clean up glue database creation
• add create database if not exists into glue-create lambda
• patch bump
• Switch the webhook to use append_jsonl like before
• missed a format, oops!
• bump the version for release
• Be a little bit more explicit in processing inbound data with sqs-ingest
• fixup! Introduce more logging for sqs-ingest to help identify schema mismatches
• Introduce more logging for sqs-ingest to help identify schema mismatches
• Rename the bootstrap.zip files to make the contents of downloaded files more obvious
• Source differently
• Run tests across freebsd/linux
• Introduce glue-create to handle initial table creation with AWS Athena
• Update the GitHub Action to include glue-sync
• Introduce a glue table updating lambda which is trigged by S3 Event Notifications
• Add the scaffolding for glue-sync
• Refactor the build scripts to all point at the same origin
• Add a simple Jenkinsfile
• Handle odd timestamp types when doing schema evolution
• Implement rudimentary schema evolution based on parquet file schema discovery
• Refactor commits to accept Actions directly and prepare for schema evolution
• Enable the group-events lambda to use UNWRAP_SNS_ENVELOPE like the others
• Add the necessary github workflow code for release of sqs-ingest
• Introduce the bulk of sqs-ingest with some refactorings for the webhook
• Create the scaffolding for sqs-ingest
• Update the webhook documentation with important settings for use
• Properly create checkpoints on every 10th commit.
• Properly checkpoint writes via the webhook lambda
• Enhance the webhook lambda to augment with a ds column for partitioning
• Update the version and add some useful debug flags
• Add release build for github actions on the webhook lambda
• Add a webhook lambda for appending JSONL
• Properly url decode keys for the auto-tag lambda
• Introduce UNWRAP_SNS_ENVELOPE which allows SNS to be introduced upstream
• Update release workflow
• Introduce the simple auto-tag Lambda for adding some tags for lifecycle policies
• Make the release supporting deletions count as 0.9.0
• Handle ObjectRemoved:Delete events and translate those into Delta table removals
• Bump to 0.8.4
• Bump version for next release
• Prevent duplicate column definitions showing up in the delta schema
• Switch the DynamoDB provisioning to on-demand for examples
• Clean up the version inclusion in the release artifacts
• Update the release workflow and clean up the REEADME for a 0.8.1 release
• Ignore s3:TestEvent in the SQS event processing pipeline
• On table creation modify the timestamp data type for simplicity's sake
• Shorten the deployment target
• Ensure that the group-id is always valid for SQS
• Introduce the meat of the group-events function
• Implement the bulk of the group-events Lamdba which will help sequence writes
• Introduce a simple GNU/Makefile to make common development tasks easier
• Fix race on lock acquisition
• Pushing common code that will be needed in the grouping lambda into the shared crate
• Restructure the workspace a bit more to pave the way for shared tooling
• Restructure project to be a workspace
• Fix missing string formatting for the lock key
• Upgrade to the latest deltalake 0.16 release
• Reload to get the latest table state before adding a checkpoint
• Iterate further on the consistency guarantees around checkpoints that oxbow can provide
• Format the lock name properly
• Re-introduce the table creation lock after delta-rs code review
• Ensure that when being asked to add redundant files oxbow does not commit
• Behavior a bit more defensively to avoid processing checkpoint parquet files
• Remove the redundant set of dynamodb locks and rely on deltalake's built-in
• Use a unique lock key per table with the deltalake library
• Pin the cc dependency which has released a very incompatible 1.0.84
• Tighten up some dependency definitions
• Fix a number of minor clippy warnings
• Upgrade to the latest deltalake 0l16.4 which includes TTL fixes for locks
• Upgrade to the latest dynamodb_lock crate
• Use a proper type for leaseDuration
• Avoid panicking when an store does not have an authentication credential
• Avoid adding redundant files to existing tables when triggered
• Refactor the locking mechanism slightly to exclude opening the table in Lambda
• Upgrade to the latest Delta Lake package
• Correct invalid test assertion
• Add some more debugging output
• Bump to version 0.3.0 for another release
• When creating a table a new table from the Lambda, list other files too
• Upgrade to the 0.15 version of the deltalake crate
• Properly push the identified partition columns into the original schema
• Upgrade the version since the lock structure changed
• Create a lock for each table as they are accessed
• Propagate the error when creating tables to ensure that the function fails
• Minor README change to trigger some automation
• Incorrect path on the release, oops!
• Remove some of cargo-dist for a simpler attempt at producing release artifacts
• Update to deltalake 0.14 to pick up some more recent improvements by the package
• You have to check out source code explicitly, oops!
• Use cargo-lambda to build properly
• Add ziglang to the builder to hopefully get the cargo-lambda tool installed
• Run the build in github actions with cargo-lambda
• Remove windows from the release process
• Remove the readme value from Cargo.toml since it doesn't yet support asciidoc
• Include sufficient metadata in Cargo.toml for a release
• Restructure the project to make it easier to use cargo-dist for releases
• Enable release from GitHub Actions
• Add the repository to Cargo.toml
• Properly checkpoint after 10 commits
• Acquire the lock with a different key than delta itself
• Modify the function to properly handle being triggered from SQS
• Convert the triggering of the lambda from S3 Event Notificaitons to an SQS queue
• Add support for locking on the table creation/append
• Ignore any records which come from inside a _delta_log directory
• Checkpoint every 10 commits.
• Url decoded keys are needed much earlier in the processing of events
• Handle url-encoded hive partition key names
• The lambda feature is required to build the lambda
• Refactor the append_to_table function into the crate
• Update the readme to include details about using the oxbow lambda function
• Properly prepare the ObjectMeta records to be added to the table
• Allow the lambda function to log to CloudWatch logs
• Introduce the main lambda functionality of creating or appending to a table
• Properly build with both features (lambda/cli)
• infer_log_path_from to help the lambda place the _delta_log correctly
• Add stub directory tree for examples using oxbow
• Scaffolding of the minimum terraform and lambda code to receive the bucket notifications
• Correct parsing error in readme
• Add an integration test for validating all the golden tables
• Clean up some suggestions from clippy
• Support creating delta tables from storage with hive style partitioning schemes
• Sync the hive/ test data with the connectors repository
• Create rust.yml
• Fix code fencing
• Add some CLI usage information to the README
• Add an integration test to perform the most simple validation of conversion
• Add a slice of the golden data set from delta-io/connectors
• Implement the most simple use-case for a command line invocation
• Move the lib functions into the lib module
• Working on discover_parquet_files() for identifying parquet files to import
• Starting implementation of the CLI version with local files for testing
• Start structuring the project to support a CLI and in the future a lambda mode
• Add deltalake with the license before starting implementation
• initial commit

Merged pull requests: #12, #13, #15, #17, #19, #24, #28