Troubleshooting

This page covers issues in dbtrail's core capture and query paths.

Stream issues

Stream not capturing changes

1. Verify the stream is active: open the console's Status view (default http://127.0.0.1:8090/) and check the stream badge says RUNNING. On a headless install, bintrail status.
2. Check that binary logging is enabled and uses row format:

   SHOW VARIABLES LIKE 'log_bin';        -- Must be ON
   SHOW VARIABLES LIKE 'binlog_format';  -- Must be ROW

3. Confirm the bintrail MySQL user has replication privileges
4. Check whether a schema filter is excluding the tables you expect: Manage servers → Edit → Schemas in the console (or --schemas / --tables on a headless bintrail stream)

Missing or incomplete changes for specific tables

If some tables show no changes or unreliable data while others work fine, check that those tables use InnoDB and have a primary key:

-- Check a specific table's engine and keys
SHOW CREATE TABLE your_schema.your_table;

• Non-InnoDB tables (for example MyISAM) are non-transactional: failed or interrupted statements can produce partial row events that dbtrail cannot reliably interpret. Convert them: ALTER TABLE your_schema.your_table ENGINE=InnoDB;
• Tables without a primary key require full table scans for row identification during binlog processing and may be ambiguous when duplicate rows exist. Add a primary key: ALTER TABLE your_schema.your_table ADD PRIMARY KEY (your_column); (replace your_column with the column or columns that uniquely identify each row).

Caution: ALTER TABLE on large tables can lock the table and take significant time. Use an online schema change tool like pt-online-schema-change or gh-ost for production databases.

See the Quick Start prerequisites for queries that find all affected tables at once.

Stream shows FAILED or "disconnected"

The stream lost the MySQL connection. The console retries it automatically with exponential backoff and shows the badge as FAILED with the error until it reattaches. Common causes:

• MySQL server restart
• Network interruption
• MySQL connection timeout

The stream will resume from its last checkpoint when the connection is restored. A stream that crash-loops for 6 hours stops retrying; press Start on the server to re-arm it after fixing the cause.

ProxySQL time-travel (Beta)

Errors you may see when running queries through the time-travel port (the MySQL wire-protocol server that handles the _flashback, _diff, and _snapshot virtual schemas). See the Time-travel SQL guide for the ways to run it, and the repository setup guide for the full ProxySQL/shim configuration these fixes refer to.

The shim emits typed MySQL wire codes so ORMs and monitoring can distinguish user-input errors from server faults. The connection stays open after any of these: the next query on the same connection can succeed once the underlying issue is resolved.

ERROR 1045 (28000): Access denied for user '<name>'@'…'

There are two authentication hops: ProxySQL authenticates your client against mysql_users, then the shim re-authenticates the same username against shim.yaml's tenants[]. Every user authorized to issue time-travel queries must appear in both.

1. Confirm your app connects with the cleartext value of mysql_password from shim.yaml
2. If shim.yaml was edited, regenerate and re-apply the ProxySQL config so the updated credentials reach the live mysql_users table:

   rm -f proxysql-setup.sql
   bintrail proxysql-config --out proxysql-setup.sql
   mysql -u admin -p -h 127.0.0.1 -P 6032 < proxysql-setup.sql

3. The shim logs the allowlisted usernames at startup; a connection from a username missing from tenants[] is rejected. Add the user to shim.yaml and restart the shim. (Older shim versions surfaced this case as unknown tenant for MySQL user "<name>".)

Query goes to MySQL instead of the shim (_flashback.t doesn't exist)

The ProxySQL query rule isn't matching, so the statement hit your real MySQL, which has no _flashback schema. Inspect the routing:

mysql -u admin -p -h 127.0.0.1 -P 6032 \
  -e "SELECT rule_id, match_pattern, destination_hostgroup FROM runtime_mysql_query_rules WHERE rule_id BETWEEN 990001 AND 990006;"

You should see six rows targeting hostgroup 991. If they're missing, re-apply proxysql-setup.sql. If they're present but the query still goes to MySQL, check that no operator rule with a smaller rule_id intercepts _flashback.* first: ProxySQL evaluates rules in rule_id order.

connection refused on the shim's port

bintrail shim isn't running, or it's listening on a different address than expected (default 127.0.0.1:3308):

systemctl status bintrail-shim
ss -tlnp | grep 3308

If the shim is dead, journalctl -u bintrail-shim -n 100 shows why. Common causes: missing or unreadable shim.yaml, a missing index DSN (--index-dsn is required on the CLI; the shim does not read it from yaml), or a mysql_password value that's not a valid YAML string (quote it).

ERROR 1064 (42000): query shape rejected

The query mentions a virtual schema but doesn't match a supported shape. The error message names the specific issue:

• Missing AS OF (for _flashback / _snapshot) or BETWEEN (for _diff), or an unparseable timestamp
• No schema selected: issue USE <database>; first, or fully qualify the table
• The WHERE column is not the table's primary key, the table has a composite primary key, or the table has no primary key in the indexed snapshot
• A JOIN between a virtual schema and a real table in the same statement: the shim has no path to fetch real-schema rows. Run two queries client-side and join in your application, or materialize the virtual side first. Federated planning is on the post-beta roadmap.

A PK predicate is not required: a _flashback / _snapshot query with no WHERE runs full-table reconstruction, subject to the row cap below.

ERROR 1235: this server only handles _flashback / _snapshot / _diff virtual-schema queries

A non-virtual-schema query reached the shim, typically a direct connection to the shim's port that bypasses ProxySQL, or misconfigured hostgroup routing. Point your client at ProxySQL, not the shim.

ERROR 1526 (HY000): time range outside retention

The AS OF instant or BETWEEN range falls outside what the index retains: the data rotated out of MySQL with no archive coverage. Narrow the time range, or check archive coverage (archive_state) and the shim's --allow-gaps flag (default is strict: fail loudly rather than silently skip gaps).

ERROR 1104 (42000): full-table reconstruction row cap

A full-table _flashback / _snapshot query returned more than 100,000 rows (the buffered-reconstruction cap during beta). Narrow the AS OF, or add WHERE <pk> = <value> to use the point-lookup path, which is uncapped.

ERROR 1105 (HY000): internal error

A real server-side failure: index DB timeout, archive fetch outage, or a resultset-build bug. Check the shim log (journalctl -u bintrail-shim) for the underlying error and retry. If 1105s persist, capture the full message and the SQL that triggered it and open an issue.

Time-travel query returns empty

Three causes produce an empty _flashback / _snapshot resultset:

1. The row had no event at-or-before the requested timestamp.
2. The latest event at-or-before the timestamp is a DELETE: the row did not exist at AS OF.
3. A coverage gap or archive-fetch failure under --allow-gaps. Without that flag the shim returns a typed error instead (1526 for coverage gaps, 1105 for archive failures), so on the default strict configuration an empty resultset never indicates a gap.

To distinguish cases 1 and 2, query _diff for the per-PK history: a deleted row produces at least one event (including the DELETE's row_before), while a row that never existed produces zero.

Queries time out without an error

The shim does not impose its own per-query timeout; ProxySQL is the source of truth via mysql-default_query_timeout and mysql-connect_timeout_server. If your query is killed without a recognisable shim error, ProxySQL is the layer that gave up. Raise those values if your _diff window or recover range legitimately needs longer.

Getting help

• Community (free core): open an issue or start a discussion on GitHub. Include the version (shown at the bottom of the console's sidebar, or bintrail --version), what you did, and any error messages. SUPPORT.md describes what the project does and does not support.
• Paid plans: every paid plan comes with direct support: email support@dbtrail.com with the same details. See dbtrail EE & Plans for the support levels.