DBTrail Documentation

Hand your data to Spark, Trino, Athena or Snowflake as Apache Iceberg tables, without touching your database.

Only when something downstream requires Iceberg. For your own queries, Query with DuckDB reads the same files with no copy.

What DBTrail keeps, your snapshot and every change since, is only read and left untouched. The export writes a new copy as Iceberg tables in a folder you choose. DuckDB and Spark read those straight off the folder; Trino, Athena and Snowflake read them through a catalog.

Run it

Snapshots → Download → Use it where it lives → Keep it current with Iceberg gives the line for that server, with its connection and backup location filled in. Replace the stars with the index password, pick a folder, run it from cron, CI or a timer. It never runs inside the capture daemon.

bintrail export iceberg --index-dsn "..." --baseline-dir /data/baselines \
  --warehouse /data/iceberg --tables shop.orders,shop.customers

Keep it current

Three ways, each run adds only what changed:

--every 1h on the same line. It stays running and starts each run an hour after the previous one ended. Runs never overlap.
Docker: docker compose --profile iceberg-export-loop up -d iceberg-export-loop runs that loop as a service. ICEBERG_EVERY in .env sets the interval.
Cron, CI or any timer, running the line once each time.

Put the tables in AWS Glue

Athena, Spark, Trino and Snowflake find tables by name in a catalog. Add --catalog glue and an S3 location:

AWS_REGION=us-west-2 bintrail export iceberg --index-dsn "..." \
  --baseline-dir /data/baselines \
  --catalog glue --warehouse s3://my-bucket/iceberg

Each source schema becomes a Glue database, each table a Glue table. AWS credentials come from the usual chain (environment, profile, instance role). --catalog rest writes to any Iceberg REST catalog instead.

Read it

Athena: nothing to create. SELECT * FROM "shop"."orders".
Spark: Iceberg's Spark runtime and AWS bundle, with a catalog of type glue. Then SELECT * FROM glue.shop.orders.
DuckDB: iceberg_scan('/data/iceberg/shop/orders') on a local folder.
Trino and Snowflake: their own Glue support for Iceberg. Not tested by us.

Checked with Athena engine 3, Spark 3.5.5 and DuckDB 1.5.5, including rows a later run updated or deleted. The exact settings per engine are in the repository reference.

Every run after the first is small

The first run loads the whole snapshot, shown as a wide bar. The next run and the one after are thin bars: only what changed. Run it as often as you want the tables to be fresh, it picks up where it left off.

Each run reads only the changes since its last run and commits one Iceberg snapshot per table. Where it left off is stored in that commit, inside the table; nothing is written back to DBTrail.

How far behind the tables are

As far behind as the interval you schedule, plus the seconds a run takes. For tables at most five minutes behind, run it every five minutes.

Measured on 0.104.1 with a source taking 20 transactions per second: twelve runs in a row, five minutes apart, took 6 to 12 seconds each, and the tables ended each run 4 to 11 seconds behind the source.

DuckDB gets slower on a table that is exported often: about 16 seconds more per run on a 300,000-row table, while Spark read the same table in under a second. For tables exported every few minutes, read with Spark or another engine.

What a run tells you

Iceberg export summary: warehouse /data/iceberg, run at 2026-08-29T10:17:00Z. shop.orders is 12,400 rows loaded from the backup plus 318 change events folded in, verdict loaded; shop.customers is 2,051 rows plus 4 events, verdict loaded.

Nothing new: says so, writes nothing.
A table it cannot advance keeps its copy, retries next run, and does not stop the others. Exit is non-zero, so your scheduler sees it.
A run that dies halfway leaves the last complete commit readable.

Limits

A second run on the same folder is refused while one holds its lock.
Writes to a local folder, or to S3 through AWS Glue or a REST catalog. A local folder copied to S3 afterwards cannot be read by name.
MySQL and MariaDB sources; the PostgreSQL capture binary does not have it.
Tables need a primary key; unsupported column types are named in the run.

Options, type mapping, reading from each engine, every refusal: repository reference.

Export to Iceberg