Merge pull request #86 from github/more-documentation

More documentation

README.md

@@ -15,6 +15,7 @@ WORK IN PROGRESS
 Please meanwhile refer to the [docs](doc) for more information. No, really, go to the [docs](doc).
 
 - [Why triggerless](doc/why-triggerless.md)
+- [Triggerless design](doc/triggerless-design.md)
 - [Cut over phase](doc/cut-over.md)
 - [Testing on replica](doc/testing-on-replica.md)
 - [Throttle](doc/throttle.md)
@@ -43,43 +44,17 @@ It is still OK to connect `gh-ost` directly on master; you will need to confirm
 Newcomer? We think you would enjoy building trust with this tool. You can ask `gh-ost` to simulate a migration on a replica -- this will not affect data on master and will not actually do a complete migration. It will operate on a replica, and end up with two tables: the original (untouched), and the migrated. You will have your chance to compare the two and verify the tool works to your satisfaction.
 
 ```
-gh-ost --conf=.my.cnf --database=mydb --table=mytable --verbose --alter="engine=innodb" --execute --initially-drop-ghost-table --initially-drop-old-table -max-load=Threads_connected=30 --switch-to-rbr --chunk-size=2500 --exact-rowcount --test-on-replica --verbose
+gh-ost --conf=.my.cnf --database=mydb --table=mytable --verbose --alter="engine=innodb" --execute --initially-drop-ghost-table --initially-drop-old-table -max-load=Threads_running=30 --switch-to-rbr --chunk-size=2500 --exact-rowcount --test-on-replica --verbose --postpone-cut-over-flag-file=/tmp/ghost.postpone.flag --throttle-flag-file=/tmp/ghost.throttle.flag
 ```
 Please read more on [testing on replica](testing-on-replica.md)
 
-#### Executing on master
+#### Migrating a master table
 
 ```
-gh-ost --conf=.my.cnf --database=mydb --table=mytable --verbose --alter="engine=innodb" --execute --initially-drop-ghost-table --initially-drop-old-table -max-load=Threads_connected=30 --switch-to-rbr --chunk-size=2500 --exact-rowcount --verbose
+gh-ost --conf=.my.cnf --database=mydb --table=mytable --verbose --alter="engine=innodb" --initially-drop-ghost-table --initially-drop-old-table --max-load=Threads_running=30 --switch-to-rbr --chunk-size=2500 --exact-rowcount --verbose --postpone-cut-over-flag-file=/tmp/ghost.postpone.flag --throttle-flag-file=/tmp/ghost.throttle.flag [--execute]
 ```
 
-Note: "executing on master" does not mean you need to _connect_ to the master. `gh-ost` is happy if you connect to a replica; it then figures out the identity of the master and makes the connection itself.
-
-#### Notable parameters
-
-Run `gh-ost --help` to get full list of parameters. We like the following:
-
-- `--conf=/path/to/my.cnf`: file where credentials are specified. Should be in (or contain) the following format:
-
-  ```
-[client]
-user=gromit
-password=123456
-  ```
-
-- `--user`, `--password`: alternatively, supply these as arguments
-
-- `--host`, `--port`: where to connect to. `gh-ost` prefers to connect to a replica, see above.
-
-- `--exact-rowcount`: actually `select count(*)` from your table prior to migration, and heuristically maintain the updating table size while migrating. This makes for quite accurate assumption on progress. When `gh-ost` says it's `99.8%` done, it really there or very closely there.
-
-- `--execute`: without this parameter, migration is a _noop_: testing table creation and validity of migration, but not touching data.
-
-- `--initially-drop-ghost-table`, `--initially-drop-old-table`: `gh-ost` maintains two tables while migrating: the _ghost_ table (which is synced from your original table and finally replaces it) and a changelog table, which is used internally for bookkeeping. By default, it panics and aborts if it sees those tables upon startup. Provide these two params to let `gh-ost` know it's OK to drop them beforehand.
-
-  We think `gh-ost` should not take chances or make assumptions about the user's tables. Dropping tables can be a dangerous, locking operation. We let the user explicitly approve such operations.
-
-- `--test-on-replica`: `gh-ost` can be tested on a replica, without actually modifying master data. We use this for testing, and we suspect new users of this tool would enjoy checking it out, building trust in this tool, before actually applying it on production masters. Read more on [testing on replica](testing-on-replica.md).
+Note: in order to migrate a table on the master you don't need to _connect_ to the master. `gh-ost` is happy (and prefers) if you connect to a replica; it then figures out the identity of the master and makes the connection itself.
 
 ## What's in a name?
 

doc/cut-over.md

@@ -2,14 +2,15 @@
 
 The cut-over is the final major step of the migration: it's the moment where your original table is pushed aside, and the ghost table (the one we secretly altered and operated on throughout the process) takes its place.
 
-MySQL poses some limitations on how the table swap can take place. While it supports an atomic swap, it does not allow for a swap under controlled lock.
+MySQL poses some limitations on how the table swap can take place. While it supports an atomic swap, it does not allow a connection to swap tables it holds under lock.
 
-The [facebook OSC](https://www.facebook.com/notes/mysql-at-facebook/online-schema-change-for-mysql/430801045932/) tool documents this nicely. Look for **"Cut-over phase"**.
+The [facebook OSC](https://www.facebook.com/notes/mysql-at-facebook/online-schema-change-for-mysql/430801045932/) tool documents this nicely. Look for **"Cut-over phase"**. The Facebook solution uses a non-atomic swap: the original table is first renamed and pushed aside, then the ghost table is renamed to take its place. In between the two renames there's a brief period of time where your table just does not exist, and queries will fail.
 
-`gh-ost` supports various types of cut-over options:
+`gh-ost` solves this by using an atomic, two-step blocking swap: while one connection holds the lock, another attempts the atomic `RENAME`. The `RENAME` is guaranteed to not be executed prematurely by positioning a sentry table which blocks the `RENAME` operation until `gh-ost` is satisfied all is in order.
 
-- `--cut-over=safe` - this is the default value if not specified otherwise. The "safe" cut-over algorithm is depicted [here](https://github.com/github/gh-ost/issues/65). It is essentially a multi-step, locking solution, where queries block until table swapping is complete -- when all operates normally. If, for some reason, connections participating in the cut-over are killed throughout the process, the worst-case scenario is a table-outage: a brief moment where the table ceases to exist, followed by an immediate rollback.
-- `--cut-over=two-step` - this method is similar to the one taken by the facebook OSC. It's non-blocking but also non-atomic. The original table is first renames and pushed aside, then the ghost table is renamed to take its place. In between the two renames there's a brief period of time where your table just does not exist, and queries will fail.
+This solution either:
+- executes successfully, in which case the tables are swapped atomically and pending connections are blocked for a brief period of time, proceeding to operate on the newly migrated table
+- or fails, due to timeout or death of some connection, in which case we are naturally returning to pre-cut-over phase, where the original table is still in place and accessible. This releases the pending connections, which are able again to write to the table, and `gh-ost` is then able to make another attempt at the cut-over.
 
 Also note:
 - With `--migrate-on-replica` the cut-over is executed in exactly the same way as on master.

doc/interactive-commands.md

@@ -25,6 +25,7 @@ replication lag on to determine throttling
 - `throttle-control-replicas`: change list of throttle-control replicas, these are replicas `gh-ost` will cehck
 - `throttle`: force migration suspend
 - `no-throttle`: cancel forced suspension (though other throttling reasons may still apply)
+- `unpostpone`: at a time where `gh-ost` is postponing the [cut-over](cut-over.md) phase, instruct `gh-ost` to stop postponing and proceed immediately to cut-over.
 - `panic`: immediately panic and abort operation
 
 ### Examples

doc/throttle.md

@@ -32,6 +32,8 @@ Otherwise you may specify your own list of replica servers you wish it to observ
 
   Example: `--replication-lag-query="SELECT ROUND(NOW() - MAX(UNIX_TIMESTAMP(ts))) AS lag FROM mydb.heartbeat"`
 
+Note that you may dynamically change the `throttle-control-replicas` list via [interactive commands](interactive-commands.md)
+
 #### Status thresholds
 
 - `--max-load`: list of metrics and threshold values; topping the threshold of any will cause throttler to kick in.
@@ -42,6 +44,12 @@ Otherwise you may specify your own list of replica servers you wish it to observ
 
   Metrics must be valid, numeric [statis variables](http://dev.mysql.com/doc/refman/5.6/en/server-status-variables.html)
 
+#### Throttle query
+
+- When provided, the `--throttle-query` is expected to return a scalar integer. A return value `> 0` implies `gh-ost` should throttle. A return value `<= 0` implied `gh-ost` is free to proceed (pending other throttling factors).
+
+An example query could be: `--throttle-query="select hour(now()) between 8 and 17"` which implies throttling auto-starts `8:00am` and migration auto-resumes at `18:00pm`.
+
 #### Manual control
 
 In addition to the above, you are able to take control and throttle the operation any time you like.
@@ -68,7 +76,7 @@ In addition to the above, you are able to take control and throttle the operatio
 
 Any single factor in the above that suggests the migration should throttle - causes throttling. That is, once some component decides to throttle, you cannot override it; you cannot force continued execution of the migration.
 
-`gh-ost` will first check the low hanging fruits: user commanded; throttling files. It will then proceed to check replication lag, and lastly it will check for status thresholds.
+`gh-ost` will first check the low hanging fruits: user commanded; throttling files. It will then proceed to check replication lag, then status thesholds, and lastly it will check the throttle-query.
 
 The first check to suggest throttling stops the search; the status message will note the reason for throttling as the first satisfied check.
 

doc/triggerless-design.md

@@ -86,7 +86,7 @@ The Facebook solution uses an "outage", two-step rename:
 
 In between those two renames there's a point in time where the table does not exist, hence there's a "table outage".
 
-`gh-ost` solves this by using an optimistic three-step locking algorithm. It is optimistic in that if no connection gets killed throughout this process, the cut-over is locking; queries are blocking on the original table and are unblocked after the ghost table has taken its place. Should any of the participating connections get killed throughout this process, the algorithm resort to "table outage" which is then rolled back.
+`gh-ost` solves this by using a two-step algorithm that blocks writes to the table, then issues an atomic swap. It uses safety latches such that the operation either succeeds, atomically, or fails, bringing us back to pre-cut-over stage.
 
 Read more on the [cut-over](cut-over.md) documentation.
 
@@ -131,6 +131,12 @@ This is the method used by GitHub to continuously validate the tool's integrity:
 
 More to come as we make progress.
 
+### No free meals
+
+#### Increased traffic
+
+The existing tools utilize triggers to propagate data changes. `gh-ost` takes upon itself to read the data, then write it back. `gh-ost` actually prefers to read from a replica and write to the master. This implies data transfers between hosts, and certainly in/out the MySQL server daemon. At this time the MySQL client library used by `gh-ost` does not support compression, and so during a migration you can expect the full volume of a table to transfer on the wire.
+
 #### Code complexity
 
 With the synchronous, trigger based approach, the role of the migration tool is relatively small. A lot of the migration is based on the triggers doing their job within the transaction space. Issues such as rollback, datatypes, cut-over are implicitly taken care of by the database. With `gh-ost`'s asynchronous approach, the tool turns complex. It connects to the master and onto a replica; it imposes as a replicating server; it writes heartbeat events; it reads binlog data into the app to be written again onto the migrated host; it need to manage connection failures, replication lag, and more.

doc/why-triggerless.md

@@ -7,7 +7,7 @@ Existing MySQL schema migration tools:
 - [LHM](https://github.com/soundcloud/lhm)
 - [oak-online-alter-table](https://github.com/shlomi-noach/openarkkit)
 
-are all using [triggers](http://dev.mysql.com/doc/refman/5.6/en/triggers.html) to propagate live changes on your table onto a ghost/shadow table that is slowly being synchronized. The tools not all work the same: while most use a synchronous approach (all changes applied on the ghost table), the Facebook tool uses an asynchronous approach (changes are appended to a changelog table, later reviewed and appleid on ghost table).
+are all using [triggers](http://dev.mysql.com/doc/refman/5.6/en/triggers.html) to propagate live changes on your table onto a ghost/shadow table that is slowly being synchronized. The tools not all work the same: while most use a synchronous approach (all changes applied on the ghost table), the Facebook tool uses an asynchronous approach (changes are appended to a changelog table, later reviewed and applied on ghost table).
 
 Use of triggers simplifies a lot of the flow in doing a live table migration, but also poses some limitations or difficulties. Here are reasons why we choose to [design a triggerless solution](triggerless-design.md) to schema migrations.
 
@@ -27,12 +27,10 @@ We know this to be a visible overhead on very busy or very large tables.
 
 ### Triggers, locks
 
-When a table with trigger is concurrently being written two, the triggers, being in same transaction space as the incoming queries, are also executed concurrently. While concurrent queries compete for resources via locks (e.g. the `auto_increment` value), the triggers need to _simultaneously_ compete for their own locks (e.g., likewise on the `auto_increment` value on the ghost table, in a synchronous solution). The competitions are un coordinated.
+When a table with triggers is concurrently being written to, the triggers, being in same transaction space as the incoming queries, are also executed concurrently. While concurrent queries compete for resources via locks (e.g. the `auto_increment` value), the triggers need to _simultaneously_ compete for their own locks (e.g., likewise on the `auto_increment` value on the ghost table, in a synchronous solution). These competitions are non-coordinated.
 
 We have evidenced near or complete lock downs in production, to the effect of rendering the table or the entire database inaccessible due to lock contention.
 
-Some lock contention can be reduced; in the case of `auto_increment`, an `innodb_autoinc_lock_mode=2` on Row Based Replication is helpful. But this is but a single example of a locking issue, and even so we have seen lockdowns on busy tables.
-
 ### Trigger based migration, no pause
 
 The triggers are used to either apply or record ongoing changes to your original table.