DBTrail
Monitoring

Status

Where to look to know capture is running, nothing was lost, and the index disk will hold. Every stream state, what it means, what to do.

Console → Status, per server. Default address http://127.0.0.1:8090/.

The Status view: a continuity verdict banner, summary counters, coverage window with earliest and latest event, the stream's mode, binlog file and position, and the Index disk card with index size, write rate, retention, steady size, free space and how long it lasts.

Stream states

Stream badges. PENDING: started, not yet proven. RUNNING: attached and writing. STALLED: connected but no progress for 5 minutes. FAILED: retries after 15 seconds, doubling to 5 minutes, and stops after 6 hours of crash loop until you fix the cause and press Start. LOST POSITION: the source purged binlogs past the checkpoint, the gap is permanent, take a fresh snapshot.
  • Restart: every stream resumes from its checkpoint. Nothing is lost while the source still has the binlogs.
  • Crash: rows written after the last checkpoint (at most about 10 seconds) are removed and read again, so nothing is stored twice. While that cleanup runs the server's row reads CLEANING UP (since v0.83.0), the startup checklist prints the step with rows removed and duration, and the stream metrics already answer, with zeros, so alert on the age of bintrail_stream_last_flush_timestamp_seconds, not on up. On a large index it takes minutes; before v0.83.0 it showed nothing at all.
  • SIGTERM or SIGINT: streams stop, save a final checkpoint, exit.
  • Freshness: checkpoint every 10 seconds; a change is usually queryable within about 10 seconds of commit. What slows it: capacity planning.

Continuity

Continuity verdicts. ok: no gap. gap lost: events missed for good, restores through the gap cannot be complete, take a fresh snapshot. unknown: this index cannot be evaluated, treat it as not proven.

Same verdict in bintrail status --format json as continuity.status (ok, gap_lost, unknown).

Index disk

The card shows size, write rate, settled size for your retention, free space and time until full. A warn or fail grade names the cause and the fix.

A full index disk loses data

A full volume stalls the stream. Once the source purges its binlogs, the gap is permanent. Alert first: Alerts.

Also showing health: Overview (restore coverage per table), Backups (newest snapshot, scheduled results), Verification (last runs).

Headless

bintrail status --index-dsn "$IDX" --format json    # last_event_time, last_checkpoint, continuity.status
bintrail status --index-dsn "$IDX" --fail-on-gap    # non-zero on a lost or unknown gap, for cron
bintrail doctor --source-dsn "$SRC" --index-dsn "$IDX"   # prerequisites, disk, days until full

GET /api/healthz: liveness, no auth. GET /api/status: the Status view as JSON, console token required. Rotation & status reference.

On this page