From 21cf1ffdc5f67d36ad71eef4ffdab915301cfd1a Mon Sep 17 00:00:00 2001 From: Shlomi Noach Date: Tue, 27 Dec 2016 08:48:34 +0200 Subject: [PATCH] updated documentation to match --replication-lag-query deprecation --- doc/command-line-flags.md | 5 +---- doc/subsecond-lag.md | 22 +++++----------------- doc/throttle.md | 10 +--------- 3 files changed, 7 insertions(+), 30 deletions(-) diff --git a/doc/command-line-flags.md b/doc/command-line-flags.md index 1707685..d7bc97b 100644 --- a/doc/command-line-flags.md +++ b/doc/command-line-flags.md @@ -100,10 +100,7 @@ On a replication topology, this is perhaps the most important migration throttli When using [Connect to replica, migrate on master](cheatsheet.md), this lag is primarily tested on the very replica `gh-ost` operates on. Lag is measured by checking the heartbeat events injected by `gh-ost` itself on the utility changelog table. That is, to measure this replica's lag, `gh-ost` doesn't need to issue `show slave status` nor have any external heartbeat mechanism. -When `--throttle-control-replicas` is provided, throttling also considers lag on specified hosts. Measuring lag on these hosts works as follows: - -- If `--replication-lag-query` is provided, use the query, trust its result to indicate lag seconds (fraction, i.e. float, allowed) -- Otherwise, issue `show slave status` and read `Seconds_behind_master` (`1sec` granularity) +When `--throttle-control-replicas` is provided, throttling also considers lag on specified hosts. Lag measurements on listed hosts is done by querying `gh-ost`'s _changelog_ table, where `gh-ost` injects a heartbeat. See also: [Sub-second replication lag throttling](subsecond-lag.md) diff --git a/doc/subsecond-lag.md b/doc/subsecond-lag.md index 00f67cb..85ff941 100644 --- a/doc/subsecond-lag.md +++ b/doc/subsecond-lag.md @@ -2,7 +2,7 @@ `gh-ost` is able to utilize sub-second replication lag measurements. -At GitHub, small replication lag is crucial, and we like to keep it below `1s` at all times. If you have similar concern, we strongly urge you to proceed to implement sub-second lag throttling. +At GitHub, small replication lag is crucial, and we like to keep it below `1s` at all times. `gh-ost` will do sub-second throttling when `--max-lag-millis` is smaller than `1000`, i.e. smaller than `1sec`. Replication lag is measured on: @@ -10,24 +10,12 @@ Replication lag is measured on: - The "inspected" server (the server `gh-ost` connects to; replica is desired but not mandatory) - The `throttle-control-replicas` list -For the inspected server, `gh-ost` uses an internal heartbeat mechanism. It injects heartbeat events onto the utility changelog table, then reads those events in the binary log, and compares times. This measurement is by default and by definition sub-second enabled. +`gh-ost` uses an internal heartbeat mechanism. It injects heartbeat events onto the utility changelog table, then reads those events in the binary log, and compares times. This measurement is on by default and by definition supports sub-second resolution. You can explicitly define how frequently will `gh-ost` inject heartbeat events, via `heartbeat-interval-millis`. You should set `heartbeat-interval-millis <= max-lag-millis`. It still works if not, but loses granularity and effect. -On the `throttle-control-replicas`, `gh-ost` only issues SQL queries, and does not attempt to read the binary log stream. Perhaps those other replicas don't have binary logs in the first place. +`gh-ost` will use its own internal heartbeat to measure lag both on the inspected replica as well as on the `--throttle-control-replicas` list, if provided. -The standard way of getting replication lag on a replica is to issue `SHOW SLAVE STATUS`, then reading `Seconds_behind_master` value. But that value has a `1sec` granularity. +In earlier versions, the `--throttle-control-replicas` list was subjected to `1` second resolution or to 3rd party heartbeat injections such as `pt-heartbeat`. This is no longer the case. The argument `--replication-lag-query` has been deprecated and no longer needed. -To be able to throttle on your production replicas fleet when replication lag exceeds a sub-second threshold, you must provide with a `replication-lag-query` that returns a sub-second resolution lag. - -As a common example, many use [pt-heartbeat](https://www.percona.com/doc/percona-toolkit/2.2/pt-heartbeat.html) to inject heartbeat events on the master. You would issue something like: - - /usr/bin/pt-heartbeat -- -D your_schema --create-table --update --replace --interval=0.1 --daemonize --pid ... - -Note `--interval=0.1` to indicate `10` heartbeats per second. - -You would then provide - - gh-ost ... --replication-lag-query="select unix_timestamp(now(6)) - unix_timestamp(ts) as ghost_lag_check from your_schema.heartbeat order by ts desc limit 1" - -Our production migrations use sub-second lag throttling and are able to keep our entire fleet of replicas well below `1sec` lag. +Our production migrations use sub-second lag throttling and are able to keep our entire fleet of replicas well below `1sec` lag. We use `--heartbeat-interval-millis=100` on our production migrations with a `--mas-lag-millis` value of between `300` and `500`. diff --git a/doc/throttle.md b/doc/throttle.md index 94e12ab..bc4a315 100644 --- a/doc/throttle.md +++ b/doc/throttle.md @@ -28,15 +28,7 @@ Otherwise you may specify your own list of replica servers you wish it to observ - `--max-lag-millis`: maximum allowed lag; any controlled replica lagging more than this value will cause throttling to kick in. When all control replicas have smaller lag than indicated, operation resumes. -- `--replication-lag-query`: `gh-ost` will, by default, issue a `show slave status` query to find replication lag. However, this is a notoriously flaky value. If you're using your own `heartbeat` mechanism, e.g. via [`pt-heartbeat`](https://www.percona.com/doc/percona-toolkit/2.2/pt-heartbeat.html), you may provide your own custom query to return a single decimal (floating point) value indicating replication lag. - - Example: `--replication-lag-query="SELECT UNIX_TIMESTAMP() - MAX(UNIX_TIMESTAMP(ts)) AS lag FROM mydb.heartbeat"` - - We encourage you to use [sub-second replication lag throttling](subsecond-lag.md). Your query may then look like: - - `--replication-lag-query="SELECT UNIX_TIMESTAMP(6) - MAX(UNIX_TIMESTAMP(ts)) AS lag FROM mydb.heartbeat"` - -Note that you may dynamically change both `replication-lag-query` and the `throttle-control-replicas` list via [interactive commands](interactive-commands.md) +Note that you may dynamically change both `--max-lag-millis` and the `throttle-control-replicas` list via [interactive commands](interactive-commands.md) #### Status thresholds