Using Oxbow with Airbyte's S3 Output Connector

Airbyte is an open-source data integration platform that enables teams to synchronize data from various sources to destinations. The Airbyte S3 connector outputs data in Parquet format, making it an excellent source for Oxbow to convert into Delta Lake tables.

Integration Architecture

Airbyte Source Connector: Extracts data from databases, APIs, or other sources
S3 Output Connector: Writes the extracted data to S3 in Parquet format
Oxbow Lambda: Processes the Parquet files and converts them to Delta Lake format
Delta Lake Table: Makes the data immediately available for analytics

Benefits of This Approach

Simplified Data Pipeline: Remove the need for custom ETL code
Schema Evolution: Both Airbyte and Oxbow handle schema changes gracefully
Incremental Syncs: Airbyte's change data capture minimizes data volume
Open Source: Full transparency and customization capabilities
Cost Effective: Pay only for the compute and storage you use

Supported Airbyte Sources

Airbyte supports hundreds of data sources including:

Databases: PostgreSQL, MySQL, MongoDB, SQL Server
Cloud Platforms: Salesforce, Google Analytics, Facebook Ads
SaaS Applications: Stripe, Shopify, HubSpot, Zendesk
Data Warehouses: Snowflake, BigQuery, Redshift
Custom Sources: Build your own connectors for specialized data sources

Implementation Guide

Step 1: Configure Airbyte S3 Destination

In your Airbyte configuration:

{
  "destinationSector": "s3",
  "destinationName": "S3",
  "destinationId": "a3f6e01c-0ff3-423f-bfb8-40856660114a",
  "configuration": {
    "bucket_name": "my-airbyte-bucket",
    "bucket_path": "airbyte-data/",
    "bucket_region": "us-west-2",
    "format": {
      "format_type": "Parquet",
      "compression": "gzip"
    },
    "output_schema_token": {
      "namespace": "public",
      "name": "airbyte"
    }
  }
}

Key configuration settings:

format.format_type: Must be "Parquet" for Oxbow compatibility
bucket_path: Organize data with appropriate prefixes
compression: GZIP or ZSTD for best performance

Step 2: Deploy Oxbow for Airbyte Integration

Using Terraform from the Oxbow repository:

cd oxbow/deployment
airbyte=cargo lambda build --release --output-format zip --bin oxbow-lambda
airbyte=terraform init
airbyte=terraform plan -var="schema_evolution=true"
airbyte=terraform apply

Step 3: Configure S3 Event Notifications

Set up automatic triggering when Airbyte writes new Parquet files:

resource "aws_s3_bucket_notification" "airbyte_oxbow" {
  bucket = "my-airbyte-bucket"

  lambda_function {
    lambda_function_arn = aws_lambda_function.oxbow.arn
    events              = ["s3:ObjectCreated:*"]
    filter_prefix       = "airbyte-data/"
    filter_suffix       = ".parquet"
  }
}

Typical Use Cases

Data Warehouse Replication

Source: Salesforce, Shopify, or other SaaS platforms
Process: Airbyte extracts data using APIs and writes to S3
Result: Delta Lake tables updated continuously with latest business data

Analytics Data Consolidation

Source: Multiple databases and APIs across the organization
Process: Airbyte centralizes all data sources in S3
Result: Unified analytics dataset in Delta Lake format

Data Lake Population

Source: Transactional databases (PostgreSQL, MySQL)
Process: Airbyte performs incremental syncs to capture changes
Result: Complete historical record in Delta Lake with minimal data transfer

Configuration Best Practices

Airbyte Configuration

Sync Frequency: Set appropriate sync intervals based on data freshness requirements
Incremental Sync: Use cursor-based or CDC where possible to minimize data transfer
Schema Management: Configure schema evolution handling appropriately
Compression: Use GZIP for best balance of compression ratio and performance

Oxbow Configuration

Schema Evolution: Enable SCHEMA_EVOLUTION environment variable for changing data structures
Logging Level: Set RUST_LOG=info,oxbow=debug for comprehensive logging
Concurrency: Configure Lambda concurrency based on expected data volume
Lock Table: Ensure DynamoDB lock table is appropriately provisioned

S3 Configuration

Bucket Structure: Organize data by source system and sync time
my-bucket/airbyte/postgres/year=2024/month=01/day=15/
Lifecycle Policies: Configure appropriate retention and tiering policies
Access Control: Set up fine-grained IAM policies for security

Data Flow Example

graph TD
    A[PostgreSQL Database] -->|Airbyte| B[Airbyte Worker]
    A2[Salesforce] -->|Airbyte| B
    A3[Google Analytics] -->|Airbyte| B
    B -->|Parquet Files| C[S3 Bucket]
    C -->|Event Notification| D[Oxbow Lambda]
    D -->|Delta Conversion| E[Delta Lake Table]
    E --> F[Athena/Trino Queries]
    E --> G[Databricks Analytics]

Performance Considerations

Data Volume

File Size: Airbyte creates files targeting 200MB compressed size
Batch Size: Oxbow processes files individually upon S3 creation events
Throughput: System scales with number of files, not individual file size

Resource Allocation

Lambda Memory: Minimum 1024MB, Recommend 2048MB for larger files
Lambda Timeout: 30-60 seconds typically sufficient for most files
DynamoDB: Ensure lock table has adequate provisioned capacity

Monitoring Setup

Airbyte Monitoring: Track sync success rates and data volumes
Lambda Monitoring: Monitor execution times and error rates
S3 Monitoring: Track object creation rates and storage usage
Delta Monitoring: Verify table updates and query performance

Migrating from Other ETL Platforms

From Custom ETL Pipelines

Identify data sources: Inventory all current data sources
Configure Airbyte connectors: Set up appropriate source connectors
Test syncs: Verify data extraction works correctly
Deploy Oxbow: Set up Delta conversion pipeline
Cutover: Switch from custom pipeline to Airbyte/Oxbow

From Commercial ETL Tools

Compare connectors: Identify equivalent Airbyte connectors
Replicate configurations: Match source configurations in Airbyte
Test incremental syncs: Ensure change data capture works properly
Monitor performance: Compare data volumes and processing times

Advanced Patterns

Multiple Delta Tables

Create separate Oxbow Lambda functions for different data sources:

# Terraform modules for different data sources
module "postgres_delta" {
  source = "./oxbow-module"
  table_prefix = "postgres_"
  s3_prefix = "airbyte/postgres/"
}

module "salesforce_delta" {
  source = "./oxbow-module"
  table_prefix = "salesforce_"
  s3_prefix = "airbyte/salesforce/"
}

Schema Validation

Add schema validation step before Delta conversion:

# Lambda preprocessing function
import pyarrow.parquet as pq


def validate_schema(filepath, expected_schema):
    """Validate Parquet file schema matches expectations"""
    table = pq.read_table(filepath)
    actual_schema = table.schema
    
    # Validate schema compatibility
    if not schemas_compatible(expected_schema, actual_schema):
        raise ValueError(f"Schema mismatch: {filepath}")
    
    return True

Conclusion

The combination of Airbyte's comprehensive data extraction capabilities with Oxbow's seamless Delta Lake conversion provides a powerful, open-source solution for building and maintaining your data lake. This architecture eliminates the need for custom ETL code while providing enterprise-grade reliability and performance.

Buoyant Data

Airbyte S3 Output to Delta Lake