Capacity Planning

dbtrail runs a lightweight Go agent on AWS Graviton EC2 instances. The agent orchestrates binlog streaming, indexing, and archival by delegating heavy work to the bintrail CLI — the agent itself is a thin HTTP server that manages subprocesses. This page covers what you need to size and plan for in production.

Instance specifications

Each plan tier runs on a specific EC2 instance type. Free tier shares an instance with other tenants; paid tiers get a dedicated instance.

Agent resource usage

The Go agent itself is minimal. CPU and memory are dominated by the bintrail CLI subprocesses and the MySQL index database, not the agent HTTP server.

Real-world measurements

These numbers come from a production demo running on a t4g.medium (2 vCPU, 4 GB RAM) indexing a WordPress database doing ~2,500 writes/day with binlog_row_image=FULL:

The remaining ~2 GB of used memory is the InnoDB buffer pool for the local index database. The agent processes themselves are lightweight.

CPU: 7–9% average across all hours of the day, with occasional spikes to ~50% during archive rotation (Parquet export + S3 upload). The 51% spike lasts seconds, not minutes. System load average holds steady at 0.1–0.2.

Connections to your source MySQL

The agent opens a small, fixed number of connections to your monitored database:

Plan limits

Rate limits apply to API calls (query, recover, status, forensics). They do not throttle the binlog stream itself, which runs continuously.

Index size and disk usage

The index database stores one row per binlog change event, including structured before/after column values, timestamps, binlog positions, and schema metadata.

How much disk does the index use?

From our production demo (WordPress, ~2,500 writes/day, binlog_row_image=FULL):

The index-to-binlog ratio depends on your row width and binlog_row_image setting. With FULL row images (which store complete before/after rows), the index is roughly 50–80% the size of the raw binlog. With MINIMAL row images, the index can be significantly smaller since only changed columns are recorded.

Sizing recommendation: start a stream, let it run for 24–48 hours, then check disk usage to extrapolate. Schema and table filtering can significantly reduce index size if you only need to track specific tables.

Retention and archiving

Retention by plan

How retention works

The rotate daemon runs on a configurable interval (default: every hour). It:

1. Exports events older than the retention window to Parquet files, partitioned by date and hour
2. Uploads Parquet files to S3 with checksum verification (size + SHA-256)
3. Deletes local Parquet files only after S3 verification succeeds
4. Purges expired rows from the live index

Archive compression

From 19 days of production data:

Parquet's columnar format with built-in compression makes long-term storage extremely efficient. Even with 365 days of retention on a moderately active server, S3 storage costs are typically under $1/month.

Indexing lag

Indexing lag is the delay between when a change occurs on your MySQL server and when it appears in the dbtrail index. It's the sum of binlog replication delay, event parsing time, and index write time.

How lag is exposed

The stream status endpoint returns two lag metrics:

• lag_seconds — time difference between the most recent indexed event and the current time
• lag_events — number of binlog events received but not yet written to the index

Both are visible in the dashboard's stream status panel and via the /api/v1/status API endpoint.

Checkpoint mechanism

The stream writes a checkpoint (binlog file + position, or GTID set) to the index database at a configurable interval. On restart, the stream resumes from the last checkpoint — no events are reprocessed or lost, as long as the source MySQL hasn't purged the binlog files covering the gap.

What affects lag

• Binlog event volume — high-throughput workloads (bulk INSERTs, batch UPDATEs) produce more events per second, increasing parsing time
• Row width — wide rows with large TEXT or BLOB columns take longer to parse and write to the index
• Network latency — relevant if the agent connects to the source MySQL over a network (not localhost) or through an SSH tunnel
• Index DB write performance — gp3 EBS volumes with baseline 3,000 IOPS handle typical workloads easily; extremely write-heavy servers may benefit from provisioned IOPS

Under typical OLTP workloads (hundreds to low thousands of writes per second), expect sub-second lag. High-throughput batch operations may temporarily increase lag, which recovers once the burst subsides.

Failure and recovery

What happens when the agent restarts?

The agent persists stream state to disk (a JSON state file in Docker mode, or systemd journal in systemd mode). On restart:

1. The agent reads the persisted state and re-launches all previously running streams
2. Each stream reads its last checkpoint from the index database
3. Streaming resumes from the checkpoint position — no events are duplicated or lost

If the source MySQL has purged the binlog files that cover the gap between the last checkpoint and the current position, a fresh snapshot (full dump) is needed to re-establish a baseline. This is the same recovery model as MySQL replication.

Auto-restart behavior

If a stream process crashes, the agent restarts it automatically:

If the stream runs for 2+ minutes after a restart, the retry counter resets — it's treated as a transient crash. After 5 consecutive failures within the stable threshold, the agent stops retrying and reports the stream as failed.

Graceful shutdown

On SIGTERM, the agent stops all streams gracefully, writes final state to disk, and exits cleanly. Systemd and Docker both send SIGTERM by default on stop/restart operations.

Monitoring

Health endpoint

GET /health — lightweight, unauthenticated. Suitable for load balancer health checks or external monitoring.

{
  "status": "healthy",
  "agent_version": "0.4.1",
  "mysql_index": "connected",
  "disk_usage_percent": 12,
  "disk_total_gb": 47.3,
  "disk_free_gb": 41.8,
  "uptime_seconds": 5422
}

Status is healthy when the index database is reachable and disk usage is below 95%. It degrades to degraded otherwise.

Stream status endpoint

GET /api/v1/status — returns detailed stream metrics including lag, checkpoint position, and schema coverage. Requires authentication via service token.

Recommended monitoring setup

Poll /health every 30–60 seconds from your monitoring system (Datadog, Prometheus, Nagios, etc.). Alert on:

• status != healthy
• disk_usage_percent > 80% (warning) or > 90% (critical)
• disk_free_gb < 5 GB

For stream-level monitoring, poll /api/v1/status and alert on lag_seconds exceeding your tolerance (e.g., > 60 seconds sustained).

Sizing recommendations

These are starting points. Actual requirements depend on row width, change frequency, and how aggressively you filter schemas and tables. Start with a trial and monitor disk usage for 48 hours before committing to a plan.

Next steps

• Stream configuration — configure binlog streaming, filtering, and checkpoints
• Backup strategy — periodic full dumps complement binlog streaming
• Troubleshooting — common issues and resolution steps
• Status API — endpoint details for monitoring integration