Merge pull request #58 from github/more-docs-3

moar, moar docs. Explaining why triggerless

README.md

@@ -14,6 +14,15 @@ WORK IN PROGRESS
 
 Please meanwhile refer to the [docs](doc) for more information. No, really, go to the [docs](doc).
 
+- [Why triggerless](doc/why-triggerless.md)
+- [Cut over phase](doc/cut-over.md)
+- [Testing on replica](doc/testing-on-replica.md)
+- [Throttle](doc/throttle.md)
+- [Migrating with Statement Based Replication](doc/migrating-with-sbr.md)
+- [Understanding output](doc/understanding-output.md)
+- [Interactive commands](doc/interactive-commands.md)
+- [Command line flags](doc/command-line-flags.md)
+
 ## Usage
 
 #### Where to execute

doc/throttle.md

@@ -80,3 +80,15 @@ The throttle status is printed as part of the periodic [status message](understa
 Copy: 0/2915 0.0%; Applied: 0; Backlog: 0/100; Elapsed: 41s(copy), 41s(total); streamer: mysql-bin.000551:47983; ETA: throttled, flag-file
 Copy: 0/2915 0.0%; Applied: 0; Backlog: 0/100; Elapsed: 42s(copy), 42s(total); streamer: mysql-bin.000551:49370; ETA: throttled, commanded by user
 ```
+
+### How long can you throttle for?
+
+Throttling time is limited by the availability of the binary logs. When throttling begins, `gh-ost` suspends reading the binary logs, and expects to resume reading from same binary log where it paused.
+
+Your availability of binary logs is typically determined by the [expire_logs_days](dev.mysql.com/doc/refman/5.6/en/server-system-variables.html#sysvar_expire_logs_days) variable. If you have `expire_logs_days = 10` (or check `select @@global.expire_logs_days`), then you should be able to throttle for up to `10` days.
+
+Having said that, throttling for so long is far fetching, in that the `gh-ost` process itself must be kept alive during that time; and the amount of binary logs to process once it resumes will potentially take days to replay.
+
+It is worth mentioning that some deployments have external scheduled scripts that purge binary logs, regardless of the `expire_logs_days` configuration. Please verify your own deployment configuration.
+
+To clarify, you only need to keep binary logs on the single server `gh-ost` connects to.

doc/why-triggerless.md

@@ -0,0 +1,62 @@
+# Why triggerless?
+
+Existing MySQL schema migration tools:
+
+- [pt-online-schema-change](https://www.percona.com/doc/percona-toolkit/2.2/pt-online-schema-change.html)
+- [Facebook OSC](https://www.facebook.com/notes/mysql-at-facebook/online-schema-change-for-mysql/430801045932/)
+- [LHM](https://github.com/soundcloud/lhm)
+- [oak-online-alter-table](https://github.com/shlomi-noach/openarkkit)
+
+are all using [triggers](http://dev.mysql.com/doc/refman/5.6/en/triggers.html) to propagate live changes on your table onto a ghost/shadow table that is slowly being synchronized. The tools not all work the same: while most use a synchronous approach (all changes applied on the ghost table), the Facebook tool uses an asynchronous approach (changes are appended to a changelog table, later reviewed and appleid on ghost table).
+
+Use of triggers simplifies a lot of the flow in doing a live table migration, but also poses some limitations or difficulties. Here are reasons why we choose to [design a triggerless solution](triggerless-design.md) to schema migrations.
+
+
+
+## Overview
+
+Triggers are stored routines which are invoked on a per-row operation upon `INSERT`, `DELETE`, `UPDATE` on a table.
+They were introduced in MySQL `5.0`.
+A trigger may contain a set of queries, and these queries run in the same transaction space as the query that manipulates the table. This makes for an atomicy of both the original operation on the table and the trigger-invoked operations.
+
+### Triggers, overhead
+
+A trigger in MySQL is a stored routine. MySQL stored routines are interpreted, never compiled. With triggers, for every `INSERT`, `DELETE`, `UPDATE` on our often busy table, we pay the necessary price of the additional write (onto ghost or changelog table), but also the price of interpreting the trigger body.
+
+We know this to be a visible overhead on very busy or very large tables.
+
+### Triggers, locks
+
+When a table with trigger is concurrently being written two, the triggers, being in same transaction space as the incoming queries, are also executed concurrently. While concurrent queries compete for resources via locks (e.g. the `auto_increment` value), the triggers need to _simultaneously_ compete for their own locks (e.g., likewise on the `auto_increment` value on the ghost table, in a synchronous solution). The competitions are un coordinated.
+
+We have evidenced near or complete lock downs in production, to the effect of rendering the table or the entire database inaccessible due to lock contention.
+
+Some lock contention can be reduced; in the case of `auto_increment`, an `innodb_autoinc_lock_mode=2` on Row Based Replication is helpful. But this is but a single example of a locking issue, and even so we have seen lockdowns on busy tables.
+
+### Trigger based migration, no pause
+
+The triggers are used to either apply or record ongoing changes to your original table.
+
+Existing online migration tools (notably `pt-online-schema-change`) support the notion of _throttling_: suspending execution when the master becomes too busy, or replication is unable to catch up.
+
+However this throttling is partial. The tool may only suspend the tedious and long task of copying the millions of rows from the original table onto the ghost table, but the tool may not, at any stage, cancel the triggers. Cancelling the triggers means loss of information, leading to incorrect data.
+
+Thus, triggers must keep operating. On busy servers, we have seen that even as the online operation throttles, the master is brought down by the load of the triggers.
+
+Read more about [`gh-ost` throttling](throttle.md)
+
+### Trigger based migration, no reliable production test
+
+We sometimes wish to experiment with a migration, or know in advance how much time it would take. A trigger-based solution allows us to run a migration on a replica, provided it uses Statement Based Replication.
+
+With Row Based Replication there is no option at all to use triggers on a replica, since the triggers only execute on the master (replica only gets notified of the _effect_ of the trigger).
+
+But even with Statement Based Replication we do not get a reliable representation of the migration as it would have executed on the master. MySQL replication (up to and including `5.6`) is single threaded, given a table, and even in `5.7` we do not expect to find concurrency on a single table. The triggers on the replica will invoke sequentially ; they will not simulate the same load, and they will not suffer from the same concurrency and locking problems depicted above, as on the master.
+
+### Trigger based migration, bound to server
+
+The trigger space is within a MySQL service. It cannot go beyond that space. We are working towards a multi-server solution. More discussion as we make progress.
+
+## Triggerless design
+
+Proceed to read about the [triggerless design](triggerless-design.md)