DBTrail

Documentation · Open source, Apache-2.0

Run your reports on an analytical replica,
not on your production MySQL.

A copy of your tables made for reports, as Parquet files in a folder or an S3 bucket you own. DBTrail follows every change in your MySQL, updates the copy on the schedule you set, and you query it with DuckDB, a free SQL tool. Your MySQL never sees those queries.

Quick Start

Already running DBTrail? Query your copy with DuckDBUndo a changeSchedule the copy

Your MySQL sends its binlog to DBTrail, which reads it from outside with nothing installed in MySQL. On the schedule you set, DBTrail updates a copy of your tables as Parquet files in a folder or S3 bucket you own. You query that copy with DuckDB, so your reports never run on MySQL.

Reports stay off MySQLReports run on the copy. The first copy reads all your tables.

Nothing to build in betweenOne install. No pipeline or scripts to write.

Updates itself on your scheduleOff until you turn it on. Every few minutes, or once a day.

Your own copyYour tables as files, in a folder you own. S3 too, if you want.

Query it with DuckDBA free SQL tool. DBTrail gives you a view per table.

What you get

reports MYSQLD · APP InnoDB BUFFER POOL · LOCKS your transactions DuckDB 3 joins · GROUP BY over the copy own process OUTSIDE MYSQLD

Reports stay off your MySQL

Your report queries run on the copy, never on MySQL. DBTrail follows the changes from outside, the way MySQL replication does. Nothing is installed on your server. The first copy reads all your tables.

files YOUR MYSQL orders A FOLDER YOU OWN orders.parquet customers.parquet items.parquet + your S3 bucket

Your own copy

Your tables become files in Parquet, an open format made for reports, in a folder you own, and in your S3 bucket too if you want.

query views.sql shop.orders shop.customers shop.items DuckDB SELECT sum(total) FROM shop.orders in seconds ONE VIEW PER TABLE

Query it in DuckDB

DuckDB is a free SQL tool that runs on your laptop or a server. DBTrail gives you a views.sql file with a view per table, so you query them by name. Open the copy through that file: another tool reading the files directly can miss recent changes.

schedule by itself PICK HOW OFTEN 5 min 1 h 24 h THEN TURN IT ON

Updates itself on your schedule

Pick how often in the web page, from every few minutes to once a day, and turn it on. From then on DBTrail brings the copy up to date by itself. Left as it comes, it runs once a day.

no etl MySQL the copy DBTRAIL ETL scripts cron jobs $curl … install.sh | sh ONE COMMAND

Nothing to build in between

No ETL, the scripts and jobs teams usually write to move data out of MySQL for reports. One command installs DBTrail, and you connect your MySQL from the browser.

before you start

What it needs from your MySQL, and what it does to it

Quick StartLimitations

What it is NOT

availability your app MySQL primary READS AND WRITES the copy NO FAILOVER

Not high availability

Your app never points at the copy. Nothing fails over to it.

backup nothing history INSTALL DAY xtrabackup KEEP YOURS

Not your backup

History starts the day you install it. Keep your physical backups.

lag your schedule REQUIRED binlog ROW FULL images

Minutes behind, not seconds

It refreshes on the schedule you set. It needs a ROW binlog with full row images.

history LOCAL DISK ONLY 48 h WITH AN S3 BUCKET as long as you keep it SNAPSHOTS STAY

History needs a bucket

Without S3, row history lasts 48 hours. Snapshots stay.

All of it: Limitations.

Works with MySQL and Percona Server 8.0 and 8.4, including RDS and Aurora (Cloud SQL should work). PostgreSQL 14+ in beta and MariaDB 10.11+ capture changes too. The scheduled copy is for MySQL.

Get your copy

  1. Install. You need Docker with Compose. One command starts DBTrail, its web page and its own MySQL: Quick Start, Install.

  2. Connect your MySQL. Create your login in the page the installer opens and add your server. The page shows the SQL that creates DBTrail's user.

  3. Take the first copy and pick a schedule. On Snapshots, press Read database now. Then, in Settings, pick how often and press Turn on. Left as it comes, it runs once a day.

  4. Query it. You need DuckDB on your machine. On Snapshots, press Download, then Download the data. Unpack it and run duckdb -init views.sql inside the folder it creates:

    $ duckdb -init views.sql
-- Loading resources from views.sql
D SELECT c.tier, count(*) AS orders, round(sum(o.total), 2) AS revenue
  FROM demo.orders o
  JOIN demo.customers c ON c.id = o.customer_id
  GROUP BY c.tier ORDER BY revenue DESC;
┌──────────┬────────┬───────────────┐
│   tier   │ orders │    revenue    │
│ varchar  │ int64  │ decimal(38,2) │
├──────────┼────────┼───────────────┤
│ platinum │ 539848 │   59426197.40 │
│ gold     │ 539752 │   59353658.54 │
│ silver   │ 539056 │   59295443.48 │
│ bronze   │ 538385 │   59242795.95 │
└──────────┴────────┴───────────────┘

    A real run on a downloaded snapshot of demo data. What you downloaded is the copy at one moment. To read the copy that keeps updating: Query with DuckDB.

Every step in detail: Quick Start.

These pages cover the web interface. The command line is in the CLI and operations reference. DBTrail is open source (Apache-2.0) and self-hosted: source on GitHub.

Quick Start

Install DBTrail, connect your MySQL and run your first query on the copy in DuckDB.