Work record / Vue.ai / 2023-now
Moving data with the open-source ecosystem
Data pipelines and egress paths designed for varied sources, large payloads, with resource constraints.
- data pipelines
- Delta Lake
- stability
Context
Moving data is easy while everything behaves. The hard cases are deeply nested source shapes, wide tables with hundreds of columns, and containers with strict memory limits. I built a custom Airbyte connector to handle these cases, moving data in and out of the Vue platform.
Alongside this, I scaled Airbyte and benchmarked ingestion across databases, files, and data lakes with varying data shapes and volumes. I built a unified egress package for Delta, Parquet, Elasticsearch, CSV, and JSONL, and extended schema inference to handle unstructured objects nested inside structured records.
Other work included migrating Cosmos DB to pgvector for embedding and RAG use cases, studying change-data-capture parity across storage backends, and adding storage backend support to several open-source projects.
Approach
Rightsize the batches based on container runtime memory limits. The connector reads the Kubernetes-enforced memory limit directly from the container’s cgroup and uses it to determine how many records to batch before writing to object storage. Batch sizes are also capped at a ceiling recommended by the cloud provider and object store. A fixed batch size either wastes memory or risks exhausting it, while the cgroup limit reflects what the container is actually allowed to use. Setting the ceiling also improved write speeds.
Get change-feed behavior without requiring a change feed. Delta’s Change Data Feed (CDF) couldn’t be a requirement: asking every customer to enable it would mean a breaking change on their side, potentially forcing expensive table rewrites. When a table already has CDF enabled, the connector uses it automatically. Otherwise, I reproduce the change-detection behavior with RocksDB, tracking primary-key state across syncs to distinguish created, updated, deleted, and unchanged records. At scale, this becomes a massive sort and set operation that would consume too much memory in-process. RocksDB handles billions of keys without requiring the entire keyspace to fit in RAM.
Consider restarts as normal. Syncs can restart midway through, causing previously processed records to appear again. The connector handles these repeated records without duplicate writes, ensuring each record is written only once per sync.
Being mindful of the edges cases. Deletions, partial batches, nested values, repeated reads, and high-resource configurations are all tested explicitly. Failure paths have fallbacks rather than assuming every sync will complete under ideal conditions.
Boundaries
- Tables with CDF enabled use it. Every other table goes through the key comparison, which reads all primary keys on each sync.
- Batch size grows with the container’s memory limit until it reaches the provider’s recommended ceiling, then stays there.