describing atomic cut-over; elabrating on increased traffic

README.md

@@ -15,6 +15,7 @@ WORK IN PROGRESS
 Please meanwhile refer to the [docs](doc) for more information. No, really, go to the [docs](doc).
 
 - [Why triggerless](doc/why-triggerless.md)
+- [Triggerless design](doc/triggerless-design.md)
 - [Cut over phase](doc/cut-over.md)
 - [Testing on replica](doc/testing-on-replica.md)
 - [Throttle](doc/throttle.md)

doc/cut-over.md

@@ -2,14 +2,15 @@
 
 The cut-over is the final major step of the migration: it's the moment where your original table is pushed aside, and the ghost table (the one we secretly altered and operated on throughout the process) takes its place.
 
-MySQL poses some limitations on how the table swap can take place. While it supports an atomic swap, it does not allow for a swap under controlled lock.
+MySQL poses some limitations on how the table swap can take place. While it supports an atomic swap, it does not allow a connection to swap tables it holds under lock.
 
-The [facebook OSC](https://www.facebook.com/notes/mysql-at-facebook/online-schema-change-for-mysql/430801045932/) tool documents this nicely. Look for **"Cut-over phase"**.
+The [facebook OSC](https://www.facebook.com/notes/mysql-at-facebook/online-schema-change-for-mysql/430801045932/) tool documents this nicely. Look for **"Cut-over phase"**. The Facebook solution uses a non-atomic swap: the original table is first renamed and pushed aside, then the ghost table is renamed to take its place. In between the two renames there's a brief period of time where your table just does not exist, and queries will fail.
 
-`gh-ost` supports various types of cut-over options:
+`gh-ost` solves this by using an atomic, two-step blocking swap: while one connection holds the lock, another attempts the atomic `RENAME`. The `RENAME` is guaranteed to not be executed prematurely by positioning a sentry table which blocks the `RENAME` operation until `gh-ost` is satisfied all is in order.
 
-- `--cut-over=safe` - this is the default value if not specified otherwise. The "safe" cut-over algorithm is depicted [here](https://github.com/github/gh-ost/issues/65). It is essentially a multi-step, locking solution, where queries block until table swapping is complete -- when all operates normally. If, for some reason, connections participating in the cut-over are killed throughout the process, the worst-case scenario is a table-outage: a brief moment where the table ceases to exist, followed by an immediate rollback.
-- `--cut-over=two-step` - this method is similar to the one taken by the facebook OSC. It's non-blocking but also non-atomic. The original table is first renames and pushed aside, then the ghost table is renamed to take its place. In between the two renames there's a brief period of time where your table just does not exist, and queries will fail.
+This solution either:
+- executes successfully, in which case the tables are swapped atomically and pending connections are blocked for a brief period of time, proceeding to operate on the newly migrated table
+- or fails, due to timeout or death of some connection, in which case we are naturally returning to pre-cut-over phase, where the original table is still in place and accessible. This releases the pending connections, which are able again to write to the table, and `gh-ost` is then able to make another attempt at the cut-over.
 
 Also note:
 - With `--migrate-on-replica` the cut-over is executed in exactly the same way as on master.

doc/triggerless-design.md

@@ -86,7 +86,7 @@ The Facebook solution uses an "outage", two-step rename:
 
 In between those two renames there's a point in time where the table does not exist, hence there's a "table outage".
 
-`gh-ost` solves this by using an optimistic three-step locking algorithm. It is optimistic in that if no connection gets killed throughout this process, the cut-over is locking; queries are blocking on the original table and are unblocked after the ghost table has taken its place. Should any of the participating connections get killed throughout this process, the algorithm resort to "table outage" which is then rolled back.
+`gh-ost` solves this by using a two-step algorithm that blocks writes to the table, then issues an atomic swap. It uses safety latches such that the operation either succeeds, atomically, or fails, bringing us back to pre-cut-over stage.
 
 Read more on the [cut-over](cut-over.md) documentation.
 
@@ -131,6 +131,12 @@ This is the method used by GitHub to continuously validate the tool's integrity:
 
 More to come as we make progress.
 
+### No free meals
+
+#### Increased traffic
+
+The existing tools utilize triggers to propagate data changes. `gh-ost` takes upon itself to read the data, then write it back. `gh-ost` actually prefers to read from a replica and write to the master. This implies data transfers between hosts, and certainly in/out the MySQL server daemon. At this time the MySQL client library used by `gh-ost` does not support compression, and so during a migration you can expect the full volume of a table to transfer on the wire.
+
 #### Code complexity
 
 With the synchronous, trigger based approach, the role of the migration tool is relatively small. A lot of the migration is based on the triggers doing their job within the transaction space. Issues such as rollback, datatypes, cut-over are implicitly taken care of by the database. With `gh-ost`'s asynchronous approach, the tool turns complex. It connects to the master and onto a replica; it imposes as a replicating server; it writes heartbeat events; it reads binlog data into the app to be written again onto the migrated host; it need to manage connection failures, replication lag, and more.