Merge pull request #93 from github/documentation

Documentation

README.md

@@ -2,66 +2,80 @@
 
 #### GitHub's online schema migration for MySQL
 
-`gh-ost` allows for online schema migrations in MySQL which are:
-- Triggerless
-- Testable
-- Pausable
-- Operations-friendly
+`gh-ost` is a triggerless online schema migration solution for MySQL. It is testable and provides with pausability, dynamic control/reconfiguration, auditing, and many operational perks.
+
+`gh-ost` produces a light workload on the master throughout the migration, decoupled from the existing workload on the migrated table.
+
+It has been designed based on years of experience with existing solutions, and changes the paradigm of table migrations.
+
+![gh-ost logo](doc/images/gh-ost-logo-light-160.png)
 
 ## How?
 
-WORK IN PROGRESS
+All existing online-schema-change tools operate in similar manner: they create a _ghost_ table in the likeness of your original table, migrate that table while empty, slowly and incrementally copy data from your original table to the _ghost_ table, meanwhile propagating ongoing changes (any `INSERT`, `DELETE`, `UPDATE` applied to your table) to the _ghost_ table. Finally, at the right time, they replace your original table with the _ghost_ table.
 
-Please meanwhile refer to the [docs](doc) for more information. No, really, go to the [docs](doc).
+`gh-ost` uses the same pattern. However it differs from all existing tools by not using triggers. We have recognized the triggers to be the source of [many limitations and risks](doc/why-triggerless.md).
 
-- [Why triggerless](doc/why-triggerless.md)
-- [Triggerless design](doc/triggerless-design.md)
-- [Cut over phase](doc/cut-over.md)
-- [Testing on replica](doc/testing-on-replica.md)
-- [Throttle](doc/throttle.md)
-- [Operational perks](doc/perks.md)
-- [Migrating with Statement Based Replication](doc/migrating-with-sbr.md)
-- [Understanding output](doc/understanding-output.md)
-- [Interactive commands](doc/interactive-commands.md)
-- [Command line flags](doc/command-line-flags.md)
+Instead, `gh-ost` [uses the binary log stream](doc/triggerless-design.md) to capture table changes, and asynchronously applies them onto the _ghost_ table. `gh-ost` takes upon itself some tasks that other tools leave for the database to perform. As result, `gh-ost` has greater control over the migration process; can truly suspend it; can truly decouple the migration's write load from the master's workload.
+
+In addition, it offers many [operational perks](doc/perks.md) that make it safer, trustworthy and fun to use.
+
+![gh-ost general flow](doc/images/gh-ost-general-flow.png)
+
+## Highlights
+
+- Build your trust in `gh-ost` by testing it on replicas. `gh-ost` will issue same flow as it would have on the master, to migrate a table on a replica, without actually replacing the original table, leaving the replica with two tables you can then compare and satisfy yourself that the tool operates correctly. This is how we continuously test `gh-ost` in production.
+- True pause: when `gh-ost` [throttles](doc/throttle.md), it truly ceases writes on master: no row copies and no ongoing events processing. By throttling, you return your master to its original workload
+- Dynamic control: you can [interactively](doc/interactive-commands.md) reconfigure `gh-ost`, even as migration still runs. You may forcibly initiate throttling.
+- Auditing: you may query `gh-ost` for status. `gh-ost` listens on unix socket or TCP.
+- Control over cut-over phase: `gh-ost` can be instructed to postpone what is probably the most critical step: the swap of tables, until such time that you're comfortably available. No need to worry about ETA being outside office hours.
+
+Please meanwhile refer to the [docs](doc) for more information. No, really, read the [docs](doc).
 
 ## Usage
 
-#### Where to execute
+The [cheatsheet](doc/cheatsheet.md) has it all. You may be interested in invoking `gh-ost` in various modes:
 
-The recommended way of executing `gh-ost` is to have it connect to a _replica_, as opposed to having it connect to the master. `gh-ost` will crawl its way up the replication chain to figure out who the master is.
+- a _noop_ migration (merely testing that the migration is valid and good to go)
+- a real migration, utilizing a replica (the migration runs on the master; `gh-ost` figures out identities of servers involved. Required mode if your master uses Statement Based Replication)
+- a real migration, run directly on the master (but `gh-ost` prefers the former)
+- a real migration on a replica (master untouched)
+- a test migration on a replica, the way for you to build trust with `gh-ost`'s operation.
 
-By connecting to a replica, `gh-ost` sets up a self-throttling mechanism; feels more comfortable in querying `information_schema` tables; and more. Connecting `gh-ost` to a replica is also the trick to make it work even if your master is configured with `statement based replication`, as `gh-ost` is able to manipulate the replica to rewrite logs in `row based replication`. See [Migrating with Statement Based Replication](migrating-with-sbr.md).
+Our tips:
 
-The replica would have to use binary logs and be configured with `log_slave_updates`.
+- [Testing above all](testing-on-replica.md), try out `--test-on-replica` first few times. Better yet, make it continuous. We have multiple replicas where we iterate our entire fleet of production tables, migrating them one by one, checksumming the results, verifying migration is good.
+- For each master migration, first issue a _noop_
+- Then issue the real thing via `--execute`.
 
-It is still OK to connect `gh-ost` directly on master; you will need to confirm this by providing `--allow-on-master`. The master would have to be using `row based replication`.
+More tips:
 
-`gh-ost` itself may be executed from anywhere. It connects via `tcp` and it does not have to be executed from a `MySQL` box. However, do note it generates a lot of traffic, as it connects as a replica and pulls binary log data.
+- Use `--exact-rowcount` for accurate progress indication
+- Use `--postpone-cut-over-flag-file` to gain control over cut-over timing
+- Get familiar with the [interactive commands](doc/interactive-commands.md)
 
-#### Testing on replica
-
-Newcomer? We think you would enjoy building trust with this tool. You can ask `gh-ost` to simulate a migration on a replica -- this will not affect data on master and will not actually do a complete migration. It will operate on a replica, and end up with two tables: the original (untouched), and the migrated. You will have your chance to compare the two and verify the tool works to your satisfaction.
-
-```
-gh-ost --conf=.my.cnf --database=mydb --table=mytable --verbose --alter="engine=innodb" --execute --initially-drop-ghost-table --initially-drop-old-table -max-load=Threads_running=30 --switch-to-rbr --chunk-size=2500 --exact-rowcount --test-on-replica --verbose --postpone-cut-over-flag-file=/tmp/ghost.postpone.flag --throttle-flag-file=/tmp/ghost.throttle.flag
-```
-Please read more on [testing on replica](testing-on-replica.md)
-
-#### Migrating a master table
-
-```
-gh-ost --conf=.my.cnf --database=mydb --table=mytable --verbose --alter="engine=innodb" --initially-drop-ghost-table --initially-drop-old-table --max-load=Threads_running=30 --switch-to-rbr --chunk-size=2500 --exact-rowcount --verbose --postpone-cut-over-flag-file=/tmp/ghost.postpone.flag --throttle-flag-file=/tmp/ghost.throttle.flag [--execute]
-```
-
-Note: in order to migrate a table on the master you don't need to _connect_ to the master. `gh-ost` is happy (and prefers) if you connect to a replica; it then figures out the identity of the master and makes the connection itself.
+Also see [requirements and limitations](doc/requirements-and-limitations.md), [what if?](doc/what-if.md)
 
 ## What's in a name?
 
 Originally this was named `gh-osc`: GitHub Online Schema Change, in the likes of [Facebook online schema change](https://www.facebook.com/notes/mysql-at-facebook/online-schema-change-for-mysql/430801045932/) and [pt-online-schema-change](https://www.percona.com/doc/percona-toolkit/2.2/pt-online-schema-change.html).
 
 But then a rare genetic mutation happened, and the `s` transformed into `t`. And that sent us down the path of trying to figure out a new acronym. Right now, `gh-ost` (pronounce: _Ghost_), stands for:
-- GitHub Online Schema Translator/Transformer/Transfigurator
+- GitHub Online Schema Transmogrifier/Translator/Transformer/Transfigurator
+
+Pronounce: _ghost_
+
+## License
+
+`gh-ost` is licensed under the [MIT license](https://github.com/github/gh-ost/blob/documentation/LICENSE)
+
+`gh-ost` uses 3rd party libraries, each with their own license. These are found [here](https://github.com/github/gh-ost/tree/master/vendor).
+
+## Community
+
+`gh-ost` is released at a stable state, but with mileage to go. We are [open to pull requests](https://github.com/github/gh-ost/blob/master/.github/CONTRIBUTING.md). Please first discuss your intentions via [Issues](https://github.com/github/gh-ost/issues).
+
+We develop `gh-ost` at GitHub and for the community. We may have different priorities than others. From time to time we may suggest a contribution that is not on our immediate roadmap but which may appeal to others. 
 
 ## Authors
 

doc/cheatsheet.md

@@ -0,0 +1,123 @@
+# Cheatsheet
+
+![operation modes](images/gh-ost-operation-modes.png)
+
+
+`gh-ost` operates by connecting to potentially multiple servers, as well as imposing itself as a replica in order to streamline binary log events directly from one of those servers. There are various operation modes, which depend on your setup, configuration, and where you want to run the migration.
+
+### a. Connect to replica, migrate on master
+
+This is the mode `gh-ost` expects by default. `gh-ost` will investigate the replica, crawl up to find the topology's master, and will hook onto it as well. Migration will:
+
+- Read and write row-data on master
+- Read binary logs events on the replica, apply the changes onto the master
+- Investigates table format, columns & keys, count rows on the replica
+- Read internal changelog events (such as heartbeat) from the replica
+- Cut-over (switch tables) on the master
+
+If your master works with SBR, this is the mode to work with. The replica must be configured with binary logs enabled (`log_bin`, `log_slave_updates`) and should have `binlog_format=ROW` (`gh-ost` can apply the latter for you).
+
+However even with RBR we suggest this is the least master-intrusive operation mode.
+
+```shell
+gh-ost \
+--max-load=Threads_running=25 \
+--critical-load=Threads_running=1000 \
+--chunk-size=1000 \
+--throttle-control-replicas="myreplica.1.com,myreplica.2.com" \
+--max-lag-millis=1500 \
+--user="gh-ost" \
+--password="123456" \
+--host=replica.with.rbr.com \
+--database="my_schema" \
+--table="my_table" \
+--verbose \
+--alter="engine=innodb" \
+--switch-to-rbr \
+--allow-master-master \
+--cut-over=default \
+--exact-rowcount \
+--default-retries=120 \
+--panic-flag-file=/tmp/ghost.panic.flag \
+--postpone-cut-over-flag-file=/tmp/ghost.postpone.flag \
+[--execute]
+```
+
+With `--execute`, migration actually copies data and flips tables. Without it this is a `noop` run.
+
+
+### b. Connect to master
+
+If you don't have replicas, or do not wish to use them, you are still able to operate directly on the master. `gh-ost` will do all operations directly on the master. You may still ask it to be considerate of replication lag.
+
+- Your master must produce binary logs in RBR format.
+- You must approve this mode via `--allow-on-master`.
+
+```shell
+gh-ost \
+--max-load=Threads_running=25 \
+--critical-load=Threads_running=1000 \
+--chunk-size=1000 \
+--throttle-control-replicas="myreplica.1.com,myreplica.2.com" \
+--max-lag-millis=1500 \
+--user="gh-ost" \
+--password="123456" \
+--host=master.with.rbr.com \
+--allow-on-master \
+--database="my_schema" \
+--table="my_table" \
+--verbose \
+--alter="engine=innodb" \
+--switch-to-rbr \
+--allow-master-master \
+--cut-over=default \
+--exact-rowcount \
+--default-retries=120 \
+--panic-flag-file=/tmp/ghost.panic.flag \
+--postpone-cut-over-flag-file=/tmp/ghost.postpone.flag \
+[--execute]
+```
+
+### c. Migrate/test on replica
+
+This will perform a migration on the replica. `gh-ost` will briefly connect to the master but will thereafter perform all operations on the replica without modifying anything on the master.
+Throughout the operation, `gh-ost` will throttle such that the replica is up to date.
+
+- `--migrate-on-replica` indicates to `gh-ost` that it must migrate the table directly on the replica. It will perform the cut-over phase even while replication is running.
+- `--test-on-replica` indicates the migration is for purpose of testing only. Before cut-over takes place, replication is stopped. Tables are swapped and then swapped back: your original table returns to its original place.
+Both tables are left with replication stopped. You may examine the two and compare data.
+
+Test on replica cheatsheet:
+```shell
+gh-ost \
+  --user="gh-ost" \
+  --password="123456" \
+  --host=replica.with.rbr.com \
+  --test-on-replica \
+  --database="my_schema" \
+  --table="my_table" \
+  --verbose \
+  --alter="engine=innodb" \
+  --initially-drop-ghost-table \
+  --initially-drop-old-table \
+  --max-load=Threads_running=30 \
+  --switch-to-rbr \
+  --chunk-size=2500 \
+  --cut-over=default \
+  --exact-rowcount \
+  --serve-socket-file=/tmp/gh-ost.test.sock \
+  --panic-flag-file=/tmp/gh-ost.panic.flag \
+  --execute
+```
+
+### cnf file
+
+You may use a `cnf` file in the following format:
+
+```
+[client]
+user=gh-ost
+password=123456
+```
+
+You may then remove `--user=gh-ost --password=123456` and specify `--conf=/path/to/config/file.cnf`

doc/command-line-flags.md

@@ -2,7 +2,21 @@
 
 A more in-depth discussion of various `gh-ost` command line flags: implementation, implication, use cases.
 
-##### conf
+### allow-on-master
+
+By default, `gh-ost` would like you to connect to a replica, from where it figures out the master by itself. This wiring is required should your master execute using `binlog_format=STATEMENT`.
+
+If, for some reason, you do not wish `gh-ost` to connect to a replica, you may connect it directly to the master and approve this via `--allow-on-master`.
+
+### approve-renamed-columns
+
+When your migration issues a column rename (`change column old_name new_name ...`) `gh-ost` analyzes the statement to try an associate the old column name with new column name. Otherwise the new structure may also look like some column was dropped and another was added.
+
+`gh-ost` will print out what it thinks the _rename_ implied, but will not issue the migration unless you provide with `--approve-renamed-columns`.
+
+If you think `gh-ost` is mistaken and that there's actually no _rename_ involved, you may pass `--skip-renamed-columns` instead. This will cause `gh-ost` to disassociate the column values; data will not be copied between those columns.
+
+### conf
 
 `--conf=/path/to/my.cnf`: file where credentials are specified. Should be in (or contain) the following format:
 
@@ -12,11 +26,11 @@ user=gromit
 password=123456
   ```
 
-##### cut-over
+### cut-over
 
 Optional. Default is `safe`. See more discussion in [cut-over](cut-over.md)
 
-##### exact-rowcount
+### exact-rowcount
 
 A `gh-ost` execution need to copy whatever rows you have in your existing table onto the ghost table. This can, and often be, a large number. Exactly what that number is?
 `gh-ost` initially estimates the number of rows in your table by issuing an `explain select * from your_table`. This will use statistics on your table and return with a rough estimate. How rough? It might go as low as half or as high as double the actual number of rows in your table. This is the same method as used in [`pt-online-schema-change`](https://www.percona.com/doc/percona-toolkit/2.2/pt-online-schema-change.html).
@@ -29,20 +43,28 @@ A `gh-ost` execution need to copy whatever rows you have in your existing table
 
 While the ongoing estimated number of rows is still heuristic, it's almost exact, such that the reported  [ETA](understanding-output.md) or percentage progress is typically accurate to the second throughout a multiple-hour operation.
 
-##### execute
+### execute
 
 Without this parameter, migration is a _noop_: testing table creation and validity of migration, but not touching data.
 
-##### initially-drop-ghost-table
+### initially-drop-ghost-table
 
 `gh-ost` maintains two tables while migrating: the _ghost_ table (which is synced from your original table and finally replaces it) and a changelog table, which is used internally for bookkeeping. By default, it panics and aborts if it sees those tables upon startup. Provide `--initially-drop-ghost-table` and `--initially-drop-old-table` to let `gh-ost` know it's OK to drop them beforehand.
 
 We think `gh-ost` should not take chances or make assumptions about the user's tables. Dropping tables can be a dangerous, locking operation. We let the user explicitly approve such operations.
 
-##### initially-drop-old-table
+### initially-drop-old-table
 
 See #initially-drop-ghost-table
 
-##### test-on-replica
+### migrate-on-replica
+
+Typically `gh-ost` is used to migrate tables on a master. If you wish to only perform the migration in full on a replica, connect `gh-ost` to said replica and pass `--migrate-on-replica`. `gh-ost` will briefly connect to the master but other issue no changes on the master. Migration will be fully executed on the replica, while making sure to maintain a small replication lag.
+
+### skip-renamed-columns
+
+See `approve-renamed-columns`
+
+### test-on-replica
 
 Issue the migration on a replica; do not modify data on master. Useful for validating, testing and benchmarking. See [test-on-replica](test-on-replica.md)

doc/requirements-and-limitations.md

@@ -0,0 +1,23 @@
+# Requirements and limitations
+
+### Requirements
+
+- You will need to have one server serving Row Based Replication (RBR) format binary logs. `gh-ost` prefers to work with replicas. You may [still have your master configured with Statement Based Replication](migrating-with-sbr) (SBR).
+
+- `gh-ost` requires an account with these privileges:
+
+  - `ALTER, CREATE, DELETE, DROP, INDEX, INSERT, LOCK TABLES, SELECT, TRIGGER, UPDATE` on the database (schema) where your migrated table is, or of course on `*.*`
+  - `SUPER, REPLICATION SLAVE` on `*.*`
+
+### Limitations
+
+- Foreign keys not supported. They may be supported in the future, to some extent.
+- Triggers are not supported. they may be supported in the future.
+- The two _before_ & _after_ tables must share some `UNIQUE KEY`. Such key would be used by `gh-ost` to iterate the table.
+  - As an example, if your table has a single `UNIQUE KEY` and no `PRIMARY KEY`, and you wish to replace it with a `PRIMARY KEY`, you will need two migrations: one to add the `PRIMARY KEY` (this migration will use the existing `UNIQUE KEY`), another to drop the now redundant `UNIQUE KEY` (this migration will use the `PRIMARY KEY`).
+- The chosen migration key must not include columns with `NULL` values.
+  - `gh-ost` will do its best to pick a migration key with non-nullable columns. It will by default refuse a migration where the only possible `UNIQUE KEY` includes nullable-columns. You may override this refusal via `--allow-nullable-unique-key` but **you must** be sure there are no actual `NULL` values in those columns. Such `NULL` values would cause a data integrity problem and potentially a corrupted migration.
+- It is not allowed to migrate a table where another table exists with same name and different upper/lower case.
+  - For example, you may not migrate `MyTable` if another table called `MYtable` exists in the same schema.
+- Amazon `RDS` is probably not supported (due to `SUPER` requirement)
+- Multisource is not supported when migrating via replica. It _should_ work (but never tested) when connecting directly to master (`--allow-on-master`)

doc/what-if.md

@@ -0,0 +1,32 @@
+# What if?
+
+Technical questions and answers. This document will be updated as we go
+
+### What if I'm using Statement Based Replication?
+
+You can still migrate tables with `gh-ost`. We do that. What you will need is a replica configured with:
+
+- `log_bin`
+- `log_slave_updates`
+- `binlog_format=ROW`
+
+Thus, the replica will transform the master's SBR binlogs into RBR binlogs. `gh-ost` is happy to read the binary logs from the replica. [Read more](migrating-with-sbr.md)
+
+### What if gh-ost crashes halfway through, or I kill it?
+
+Unlike trigger-based solutions, there's nothing urgent to clean up in the event `gh-ost` bails out or gets killed. There are the two tables creates by `gh-ost`:
+
+- The _ghost_ table: `_yourtablename_gho`
+- The _changelog_ table: `_yourtablename_ghc`
+
+You may instruct `gh-ost` to drop these tables upon startup; or better yet, you drop them.
+
+### What if the cut-over (table switch) is unable to proceed due to locks/timeout?
+
+There is a `lock_wait_timeout` explicitly associated with the cut-over operation. If your table suddenly suffers from a long running query, the cut-over (involving `LOCK` and `RENAME` statements) may be unable to proceed. There's a finite number of retries, and if none of these succeeds, `gh-ost` bails out.
+
+### What if the migration is causing a high load on my master?
+
+This is where `gh-ost` shines. There is no need to kill it as you may be used to with other tools. You can reconfigure `gh-ost` [on the fly](https://github.com/github/gh-ost/blob/master/doc/interactive-commands.md) to be nicer.
+
+You're always able to actively begin [throttling](throttle.md). Just touch the `throttle-file` or `echo throttle` into `gh-ost`. Otherwise, reconfigure your `max-load`, the `nice-ratio`, the `throttle-query` to gain better thresholds that would suit your needs.

doc/why-triggerless.md

@@ -12,7 +12,6 @@ are all using [triggers](http://dev.mysql.com/doc/refman/5.6/en/triggers.html) t
 Use of triggers simplifies a lot of the flow in doing a live table migration, but also poses some limitations or difficulties. Here are reasons why we choose to [design a triggerless solution](triggerless-design.md) to schema migrations.
 
 
-
 ## Overview
 
 Triggers are stored routines which are invoked on a per-row operation upon `INSERT`, `DELETE`, `UPDATE` on a table.