Crate buoyant_kernel 

Delta Kernel

Delta-kernel-rs is an experimental Delta implementation focused on interoperability with a wide range of query engines. It supports reads and (experimental) writes (only blind appends in the write path currently). This library defines a number of traits which must be implemented to provide a working delta implementation. They are detailed below. There is a provided “default engine” that implements all these traits and can be used to ease integration work. See DefaultEngine for more information.

A full rust example for reading table data using the default engine can be found in the read-table-single-threaded example (and for a more complex multi-threaded reader see the read-table-multi-threaded example). An example for reading the table changes for a table using the default engine can be found in the read-table-changes example. The write-table example demonstrates how to write data to a Delta table using the default engine.

Engine trait

The Engine trait allows connectors to bring their own implementation of functionality such as reading parquet files, listing files in a file system, parsing a JSON string etc. This trait exposes methods to get sub-engines which expose the core functionalities customizable by connectors.

Expression handling

Expression handling is done via the EvaluationHandler, which in turn allows the creation of ExpressionEvaluators. These evaluators are created for a specific predicate Expression and allow evaluation of that predicate for a specific batch of data.

File system interactions

Delta Kernel needs to perform some basic operations against file systems like listing and reading files. These interactions are encapsulated in the StorageHandler trait. Implementers must take care that all assumptions on the behavior of the functions - like sorted results - are respected.

Reading log and data files

Delta Kernel requires the capability to read and write json files and read parquet files, which is exposed via the JsonHandler and ParquetHandler respectively. When reading files, connectors are asked to provide the context information they require to execute the actual operation. This is done by invoking methods on the StorageHandler trait.

Re-exports

pub use engine_data::EngineData;

pub use engine_data::FilteredEngineData;

pub use engine_data::FilteredRowVisitor;

pub use engine_data::GetData;

pub use engine_data::RowIndexIterator;

pub use engine_data::RowVisitor;

pub use error::DeltaResult;

pub use error::DeltaResultIterator;

pub use error::DeltaResultIteratorStatic;

pub use error::Error;

pub use expressions::Expression;

pub use expressions::ExpressionRef;

pub use expressions::Predicate;

pub use expressions::PredicateRef;

pub use snapshot::Snapshot;

pub use snapshot::SnapshotRef;

pub use delta_kernel_derive;

Modules

actions

Provides parsing and manipulation of the various actions defined in the Delta specification

checkpoint

This module implements the API for customized Delta checkpoint writes, where the caller drives the write themselves. The entry point is Snapshot::create_checkpoint_writer.

commit_range

Read a contiguous range of raw Delta commits.

committer

The committer module provides a Committer trait which allows different implementations to define how to commit transactions to a catalog or filesystem. For catalog-managed tables, a Committer specific to the managing catalog should be provided. For non-catalog-managed tables, the FileSystemCommitter should be used to commit directly to the object store (via put-if-absent call to storage to atomically write new commit files).

engine_data

Traits that engines need to implement in order to pass data between themselves and kernel.

error

Definitions of errors that the delta kernel can encounter

expressions

Definitions and functions to create and manipulate kernel expressions

history_manager

This module provides functions for performing timestamp queries over the Delta Log, translating between timestamps and Delta versions.

incremental_scan

Streams the file-action diff between two table versions. See IncrementalScanBuilder for the entry point.

kernel_predicates

Support for kernel-driven predicate evaluation via the KernelPredicateEvaluator trait. Various trait implementations are used for partition pruning, stats-based data skipping, and parquet row group filtering. The evaluation is normally performed over Scalar values, but data skipping “evaluation” actually produces a transformed predicate that replaces column references with stats column references, which log replay will instruct the engine to evaluate.

metrics

Metrics collection for Delta Kernel operations.

partition

Partition utilities for Delta table writes.

reserved_field_ids

Reserved field IDs for metadata columns in Delta tables.

scan

Functionality to create and execute scans (reads) over data stored in a delta table

schema

Definitions and functions to create and manipulate kernel schema

snapshot

In-memory representation of snapshots of tables (snapshot is a table at given point in time, it has schema etc.)

struct_patch

Struct patches: sparse, O(changes) edits to the fields of an input struct.

table_changes

Provides an API to read the table’s change data feed between two versions.

table_configuration

This module defines [TableConfiguration], a high level api to check feature support and feature enablement for a table at a given version. This encapsulates [Protocol], [Metadata], Schema, TableProperties, and ColumnMappingMode. These structs in isolation should be considered raw and unvalidated if they are not a part of [TableConfiguration]. We unify these fields because they are deeply intertwined when dealing with table features. For example: To check that deletion vector writes are enabled, you must check both both the protocol’s reader/writer features, and ensure that the deletion vector table property is enabled in the TableProperties.

table_features

table_properties

Delta Table properties. Note this module implements per-table configuration which governs how table-level capabilities/properties are configured (turned on/off etc.). This is orthogonal to protocol-level ‘table features’ which enable or disable reader/writer features (which then usually must be enabled/configured by table properties).

transaction

transforms

Shared transform infrastructure.

Macros

transform_output_type

Defines a transform’s Output and Residual associated types.

Structs

ActionReconciliationIterator

Iterator over action reconciliation data.

ActionReconciliationIteratorState

Stats for ActionReconciliationIterator

FileMeta

The metadata that describes an object.

FileSizeHistogram

Tracks the distribution of file sizes across predefined bins.

FileStats

File-level statistics for a table version: total file count, size, and histogram.

LogCompactionWriter

Writer for log compaction files

LogPath

A path to a valid delta log file. You can parse a given FileMeta into a LogPath using LogPath::try_new.

ParquetFooter

Metadata from a Parquet file footer.

Traits

AsAny

Extension trait that makes it easier to work with traits objects that implement Any, implemented automatically for any type that satisfies Any, Send, and Sync. In particular, given some trait T: Any + Send + Sync, it allows upcasting T to dyn Any + Send + Sync, which in turn allows downcasting the result to a concrete type.

DynPartialEq

Extension trait that facilitates object-safe implementations of PartialEq.

Engine

The Engine trait encapsulates all the functionality an engine or connector needs to provide to the Delta Kernel in order to read the Delta table.

EvaluationHandler

Provides expression evaluation capability to Delta Kernel.

ExpressionEvaluator

Trait for implementing an Expression evaluator.

JsonHandler

Provides JSON handling functionality to Delta Kernel.

ParquetHandler

Provides Parquet file related functionalities to Delta Kernel.

PredicateEvaluator

Trait for implementing a Predicate evaluator.

StorageHandler

Provides file system related functionalities to Delta Kernel.

Functions

should_compact

Determine if log compaction should be performed based on the commit version and compaction interval.

Type Aliases

FileDataReadResult

Data read from a Delta table file and the corresponding scan file information.

FileDataReadResultIterator

An iterator of data read from specified files

FileIndex

FileSize

FileSlice

A specification for a range of bytes to read from a file location

Version

Delta table version is 8 byte unsigned int