Capacity Planning

dbtrail's capture pipeline is deliberately lightweight: one daemon (the console's watch process in the Compose stack, or the headless bintrail up) runs the replication stream, indexer, and built-in rotation in a single process, and the heavy state lives in the MySQL index database, not in dbtrail itself. Sizing a deployment is mostly about sizing the index database and its disk. This page covers what you need to plan for in production.

Resource usage

CPU and memory are dominated by the index database (InnoDB buffer pool), not the dbtrail processes.

Real-world measurements

These numbers come from dbtrail's own production deployment, measured 2026-08-23: a single 2 vCPU / 8 GB host (AWS m7g.large) running the full Compose stack and capturing two sources at once with binlog_row_image=FULL: the WordPress database that also lives on the box (~4,500 writes/day) and an external RDS MySQL doing ~255,000 writes/day.

The dbtrail daemon itself stays under 100 MB even with a quarter-million events a day flowing through it; the index database's InnoDB buffer pool is where memory actually goes.

CPU: ~7% daily average on the 2 vCPUs, with spikes during archive rotation and baseline runs (Parquet export + S3 upload) that last seconds, not minutes. System load average holds steady at 0.1–0.2.

Connections to your source MySQL

Streaming opens a single replication connection to your monitored database (binlog streaming over the replication protocol), the same footprint as one MySQL replica.

Index size and disk usage

The index database stores one row per binlog change event, including structured before/after column values, timestamps, binlog positions, and schema metadata.

How much disk does the index use?

From the same production deployment (two sources, 1-day live retention, archives to S3):

The whole index MySQL datadir is ~3 GB, dominated by fixed InnoDB overhead (hourly partitions, redo, undo) rather than event payload. Two things drive the per-event cost: row width (a wide WordPress row costs far more than a narrow one; the RDS workload above averages ~1 KB per indexed event) and binlog_row_image (with FULL, complete before/after rows are stored; with MINIMAL, only changed columns).

Sizing recommendation: start a stream, let it run for 24–48 hours, then check disk usage to extrapolate. Schema and table filtering can significantly reduce index size if you only need to track specific tables.

Retention and archiving

Retention is fully under your control, and you tune it from the console: Settings → Rotation sets the retention window and rotation interval live: changes apply on the next cycle, no flags or restarts. The policy is one shared schedule covering every index the daemon rotates.

For archival, set the Archive to S3 field on a monitored source: rotation then uploads that source's rotated partitions as Parquet to your bucket before dropping them, so the forensic record survives the retention window and stays queryable; the console auto-discovers the archive on the next query. A persistently failing upload keeps partitions undropped and escalates to a loud error rather than silently losing data.

Headless deployments run the standalone rotate daemon instead. See the rotation & status reference.

How retention works

Rotation runs on a configurable interval (default: every hour). It:

1. Exports events older than the retention window to Parquet files, partitioned by date and hour
2. Uploads Parquet files to S3
3. Deletes local Parquet files only after the upload succeeds
4. Purges expired rows from the live index

Archive compression

From the same deployment, ~8 weeks of archives across both sources:

Parquet's columnar format with built-in compression makes long-term storage extremely efficient: zstd compression is the default (snappy and gzip are also supported). Even with a year of archives from these two servers, S3 storage costs stay under $1/month.

Indexing lag

Indexing lag is the delay between when a change occurs on your MySQL server and when it appears in the dbtrail index. It's the sum of binlog replication delay, event parsing time, and index write time.

How lag is exposed

• The web console (default 127.0.0.1:8090) shows live stream lag per server
• Prometheus. The daemon can expose a /metrics endpoint including the bintrail_stream_replication_lag_seconds gauge: seconds between now and the timestamp of the last processed event
• JSON for scripts. The console's GET /api/status (with the console token) serves the Status view's data as JSON; headless installs get the same from bintrail status --format json (per-stream last_event_time and last_checkpoint timestamps)

Checkpoint mechanism

The stream writes a checkpoint (binlog file + position, or GTID set) to the index database at a configurable interval (--checkpoint, default 10 seconds). On restart, the stream resumes from the last checkpoint: no events are reprocessed or lost, as long as the source MySQL hasn't purged the binlog files covering the gap.

What affects lag

• Binlog event volume: high-throughput workloads (bulk INSERTs, batch UPDATEs) produce more events per second, increasing parsing time
• Row width: wide rows with large TEXT or BLOB columns take longer to parse and write to the index
• Network latency: relevant if dbtrail connects to the source MySQL over a network (not localhost) or through an SSH tunnel
• Index DB write performance: SSD-backed storage handles typical workloads easily; extremely write-heavy servers may benefit from faster disks or provisioned IOPS

Under typical OLTP workloads (hundreds to low thousands of writes per second), expect sub-second lag. High-throughput batch operations may temporarily increase lag, which recovers once the burst subsides.

Failure and recovery

What happens on restart?

On startup, each stream reads its last checkpoint from the index database and resumes from that position: no events are duplicated or lost. Run the capture daemon under systemd or Docker with a restart policy (the bundled Compose stack already sets restart: unless-stopped) so the process comes back automatically.

If the source MySQL has purged the binlog files that cover the gap between the last checkpoint and the current position, a fresh snapshot (full dump) is needed to re-establish a baseline. This is the same recovery model as MySQL replication.

Auto-restart behavior

The combined daemon bintrail-console watch supervises its streams with crash-loop backoff: a stream that errors is restarted with a delay that doubles from 15 seconds up to a 5-minute cap, and the retry counter resets after a sustained healthy run. A running stream that saves no checkpoint and flushes no batch for 5 minutes is reported as stalled; a stream that has been crash-looping continuously for 6 hours is given up on (restart it explicitly, or restart the daemon, to re-arm).

Graceful shutdown

On SIGINT/SIGTERM, dbtrail stops streaming gracefully, saves a final checkpoint, and exits cleanly. Systemd and Docker both send SIGTERM by default on stop/restart operations.

Monitoring

Three built-in surfaces cover monitoring:

• Web console, the primary surface: live stream health, lag, archives, and the always-present continuity verdict, plus an unauthenticated liveness probe at /api/healthz for load balancers and uptime checks.
• Prometheus. The daemon can expose a /metrics endpoint covering all supervised streams. Key series: bintrail_stream_replication_lag_seconds, bintrail_stream_events_indexed_total, bintrail_stream_errors_total, bintrail_stream_checkpoint_saves_total. Alert on lag sustained above your tolerance (for example, > 60 seconds) and on a rising error counter. The endpoint has no built-in authentication, so bind it to an internal interface. Setup: observability.
• CLI: bintrail status --format json for cron-friendly point-in-time checks, and bintrail doctor for prerequisite and disk-capacity checks including the days-until-full projection: rotation & status reference.

Sizing recommendations

These are starting points. Actual requirements depend on row width, change frequency, and how aggressively you filter schemas and tables. Disk dominates, scaling with retention. Run a stream for 24–48 hours, then extrapolate from real usage with bintrail doctor.

Don't want to run this yourself?

Everything on this page (sizing the index database, watching its disk, retention, upgrades, and pipeline monitoring) is exactly the operational load the Managed plan takes off you: dbtrail provisions and operates the index infrastructure end-to-end while your production database stays untouched (read-only replication client, always).

Next steps

• Stream configuration: configure binlog streaming, filtering, and checkpoints
• Backup strategy: how dbtrail's backup system works (mydumper snapshots + continuous binlog streaming)
• Troubleshooting: common issues and resolution steps