We strongly believe that open source data technology is the right choice for most organizations. An open source solution allows you to take something off the shelf, and tailor it to the unique needs of your data platform. We can suggest and in some cases support your organization's adoption of open source tooling for your data and ML platform.
- delta-rs, Rust and Python bindings to Delta Lake.
- kafka-delta-ingest, high speed and efficiency ingestion for Delta Lake.
- Apache Airflow, a platform created by the community to programmatically author, schedule and monitor workflows.
- Apache Arrow (Rust), implementation of the Arrow in-memory data format in Rust.
- Apache Kafka, a distributed event streaming platform.
- DataFusion, an extensible query planning, optimization, and execution framework, written in Rust, that uses Apache Arrow as its in-memory format.
- terraform-provider-databricks, a provider for automating Databricks infrastructure with Terraform.
If you need help with open source data technology in your organization, let us know!