DBTrail Documentation

DBTrail as one Fargate task, with the index in your own MySQL, the saved state on EFS and the web interface behind a load balancer.

A browser reaches an ALB on HTTPS, which forwards to port 8090 of one Fargate task running DBTrail, capture and web interface together. Below it: EFS keeps what it saves (servers, login, snapshots); your own MySQL, for example RDS, is the index, its DSN read from a secret; the source is your database, read like a replica; the task's own disk under /tmp is where full reads stage.

One task runs what the Docker Compose stack runs: capture and the web interface. You add servers in the web interface, as with Compose.

Before you start

An index MySQL, 8.0 or newer (RDS works). Its user creates databases: bintrail_index and one bintrail_idx_<id> per server.
An EFS file system with an access point for user and group 999.
An ALB with HTTPS, and a target group of type ip, port 8090, health check /api/healthz.
The index DSN in Secrets Manager: user:password@tcp(host:3306)/bintrail_index.

Create the service once the EFS mount targets are available.

The task definition

{
  "family": "dbtrail",
  "requiresCompatibilities": ["FARGATE"],
  "networkMode": "awsvpc",
  "cpu": "1024",
  "memory": "4096",
  "runtimePlatform": { "cpuArchitecture": "ARM64", "operatingSystemFamily": "LINUX" },
  "ephemeralStorage": { "sizeInGiB": 30 },
  "executionRoleArn": "arn:aws:iam::<account>:role/<execution-role>",
  "taskRoleArn": "arn:aws:iam::<account>:role/<task-role>",
  "volumes": [{
    "name": "state",
    "efsVolumeConfiguration": {
      "fileSystemId": "<fs-id>",
      "transitEncryption": "ENABLED",
      "authorizationConfig": { "accessPointId": "<fsap-id>", "iam": "ENABLED" }
    }
  }],
  "containerDefinitions": [{
    "name": "dbtrail",
    "image": "ghcr.io/dbtrail/bintrail-console:<version>",
    "essential": true,
    "command": ["watch", "--log-format", "json"],
    "portMappings": [{ "containerPort": 8090, "protocol": "tcp" }],
    "environment": [
      { "name": "BINTRAIL_CONSOLE_LISTEN", "value": "0.0.0.0:8090" },
      { "name": "BINTRAIL_CONSOLE_ALLOWED_HOSTS", "value": "<dbtrail.example.com>" },
      { "name": "BINTRAIL_CONSOLE_ALLOW_SETUP", "value": "1" },
      { "name": "BINTRAIL_CONSOLE_SERVERS", "value": "/var/lib/bintrail/console-servers.yaml" },
      { "name": "BINTRAIL_CONSOLE_AUTH", "value": "/var/lib/bintrail/console-auth.yaml" },
      { "name": "BINTRAIL_CONSOLE_MCP_TOKEN_FILE", "value": "/var/lib/bintrail/console-mcp-token.yaml" },
      { "name": "BINTRAIL_CONSOLE_BASELINE_TRIGGER", "value": "1" },
      { "name": "BINTRAIL_CONSOLE_VERIFY_TRIGGER", "value": "1" },
      { "name": "AWS_REGION", "value": "<region>" }
    ],
    "secrets": [{ "name": "BINTRAIL_INDEX_DSN", "valueFrom": "<secret ARN>" }],
    "mountPoints": [{ "sourceVolume": "state", "containerPath": "/var/lib/bintrail" }],
    "healthCheck": {
      "command": ["CMD-SHELL", "wget -q -T 5 -O /dev/null http://127.0.0.1:8090/api/healthz"],
      "interval": 30, "timeout": 10, "retries": 3, "startPeriod": 60
    },
    "stopTimeout": 60,
    "logConfiguration": {
      "logDriver": "awslogs",
      "options": { "awslogs-group": "/ecs/dbtrail", "awslogs-region": "<region>", "awslogs-stream-prefix": "dbtrail" }
    }
  }]
}

The image is published for ARM64 and X86_64. Pin a version.
ALLOWED_HOSTS: every name people open, comma separated, no spaces.
ALLOW_SETUP: the first visitor creates the username and password. Keep the ALB closed to everyone else until then.
Full reads stage under /tmp, the task's own disk. Size ephemeralStorage to about your data, uncompressed (up to 200 GiB).
The task role needs the S3 policy if a server keeps its copy in S3; the execution role needs to read the secret.

Deployments

Run one task: desired count 1, no autoscaling. Keep the ECS default deployment, which starts the new task before stopping the old one:

A deployment with ECS default settings. The old task captures until it stops. The new task waits, showing waiting for other DBTrail, then captures from where the old one stopped. Capture pauses about 20 seconds between them. Result: 0 duplicates, every change in the index once, and 0 lost.

Measured on Fargate over three deployments, with writes going on. While both tasks run:

Sign in again if the web interface asks.
Don't change servers or settings. A change made in that minute can be lost.

On 0.99.0 and older

Set minimumHealthyPercent: 0 and maximumPercent: 100 instead, so the old task stops first. With the default, capture stops after the deployment, or both tasks store every change twice. The first upgrade from 0.99.0 needs this too.

Keep your source's binary logs longer than a deployment: on RDS, binlog retention hours of a few hours or more.

The MySQL port (Query Parquet with MySQL): it is answered by a second container in the task, which runs bintrail-console router: the task definition. Map port 3309, put an internal Network Load Balancer in front, and turn it on in the web interface. Its password is kept on EFS.
A task that dies without stopping keeps its lock until the index server notices, up to about two hours: how to free it.
Alerts: add --metrics-addr :9090 and alert when bintrail_stream_last_flush_timestamp_seconds goes stale: Alerts.
Every setting explained: ecs.md.

Run on Amazon ECS

Before you start

The task definition

Deployments

More

On this page