From a6030898131f697bfcf0ecc91ac1bffb14e3cce0 Mon Sep 17 00:00:00 2001 From: Jacob Bednarz Date: Fri, 24 Mar 2017 13:42:09 +1100 Subject: [PATCH 01/11] AWS RDS! --- doc/rds.md | 1 + 1 file changed, 1 insertion(+) create mode 100644 doc/rds.md diff --git a/doc/rds.md b/doc/rds.md new file mode 100644 index 0000000..9a1256d --- /dev/null +++ b/doc/rds.md @@ -0,0 +1 @@ +# Amazon RDS \ No newline at end of file From 8a854b8e6f03bdb8665e5db4119da9b3e06fc695 Mon Sep 17 00:00:00 2001 From: Jacob Bednarz Date: Fri, 24 Mar 2017 13:42:33 +1100 Subject: [PATCH 02/11] Add known limitations of AWS RDS --- doc/rds.md | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/doc/rds.md b/doc/rds.md index 9a1256d..cf85f49 100644 --- a/doc/rds.md +++ b/doc/rds.md @@ -1 +1,7 @@ -# Amazon RDS \ No newline at end of file +# Amazon RDS# Amazon RDS + +## Limitations + +- No `SUPER` privileges. +- `gh-ost` runs should be setup use [`--assume-rbr`][assume_rbr_docs] and use `binlog_format=ROW`. +- Aurora does not allow editing of the `read_only` parameter. While it is defined as `{TrueIfReplica}`, the parameter is non-modifiable field. From 0b93e3697ea3a906e015670a9e5433d06f13c226 Mon Sep 17 00:00:00 2001 From: Jacob Bednarz Date: Fri, 24 Mar 2017 13:44:15 +1100 Subject: [PATCH 03/11] Add notes for aurora replication Due to the way aurora is architected, replication means something slightly different to the traditional sense. This includes a work around to use test/migrate on replica instead of only master migrations. --- doc/rds.md | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/doc/rds.md b/doc/rds.md index cf85f49..3864ea9 100644 --- a/doc/rds.md +++ b/doc/rds.md @@ -5,3 +5,14 @@ - No `SUPER` privileges. - `gh-ost` runs should be setup use [`--assume-rbr`][assume_rbr_docs] and use `binlog_format=ROW`. - Aurora does not allow editing of the `read_only` parameter. While it is defined as `{TrueIfReplica}`, the parameter is non-modifiable field. + +## Aurora + +#### Replication + +In Aurora replication, you have separate reader and writer endpoints however because the cluster shares the underlying storage layer, `gh-ost` will detect it is running on the master. This becomes an issue when you wish to use [migrate/test on replica][migrate_test_on_replica_docs] because you won't be able to use a single cluster in the same way you would with MySQL RDS. + +To work around this, you can follow along the [AWS replication between clusters documentation][aws_replication_docs] for Aurora with one small caveat. For the "Create a Snapshot of Your Replication Master" step, the binlog position is not available in the AWS console. You will need to issue the SQL query `SHOW SLAVE STATUS` or `aws rds describe-events` API call to get the correct position. + +[migrate_test_on_replica_docs]: https://github.com/github/gh-ost/blob/master/doc/cheatsheet.md#c-migratetest-on-replica +[aws_replication_docs]: http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Aurora.Overview.Replication.MySQLReplication.html \ No newline at end of file From 99f7b8d8c75c1a80666592a20e680778eeb23bdf Mon Sep 17 00:00:00 2001 From: Jacob Bednarz Date: Fri, 24 Mar 2017 13:44:55 +1100 Subject: [PATCH 04/11] Add link to `pt-table-checksum` patch --- doc/rds.md | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/doc/rds.md b/doc/rds.md index 3864ea9..9490a7e 100644 --- a/doc/rds.md +++ b/doc/rds.md @@ -14,5 +14,16 @@ In Aurora replication, you have separate reader and writer endpoints however bec To work around this, you can follow along the [AWS replication between clusters documentation][aws_replication_docs] for Aurora with one small caveat. For the "Create a Snapshot of Your Replication Master" step, the binlog position is not available in the AWS console. You will need to issue the SQL query `SHOW SLAVE STATUS` or `aws rds describe-events` API call to get the correct position. +#### Percona Toolkit + +If you use `pt-table-checksum` as a part of your data integrity checks, you might want to check out [this patch][percona_toolkit_patch] which will enable you to run `pt-table-checksum` with the `--no-binlog-format-check` flag and prevent errors like the following: + +``` +03-24T12:51:06 Failed to /*!50108 SET @@binlog_format := 'STATEMENT'*/: DBD::mysql::db do failed: Access denied; you need (at least one of) the SUPER privilege(s) for this operation [for Statement "/*!50108 SET @@binlog_format := 'STATEMENT'*/"] at pt-table-checksum line 9292. + +This tool requires binlog_format=STATEMENT, but the current binlog_format is set to ROW and an error occurred while attempting to change it. If running MySQL 5.1.29 or newer, setting binlog_format requires the SUPER privilege. You will need to manually set binlog_format to 'STATEMENT' before running this tool. +``` + [migrate_test_on_replica_docs]: https://github.com/github/gh-ost/blob/master/doc/cheatsheet.md#c-migratetest-on-replica -[aws_replication_docs]: http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Aurora.Overview.Replication.MySQLReplication.html \ No newline at end of file +[aws_replication_docs]: http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Aurora.Overview.Replication.MySQLReplication.html +[percona_toolkit_patch]: https://github.com/jacobbednarz/percona-toolkit/commit/0271ba6a094da446a5e5bb8d99b5c26f1777f2b9 \ No newline at end of file From fb3993e56031d9f895e931f827b8a1a4cc05cfcd Mon Sep 17 00:00:00 2001 From: Jacob Bednarz Date: Fri, 24 Mar 2017 13:45:01 +1100 Subject: [PATCH 05/11] Fix line endings --- doc/rds.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/doc/rds.md b/doc/rds.md index 9490a7e..baec8fb 100644 --- a/doc/rds.md +++ b/doc/rds.md @@ -1,8 +1,8 @@ -# Amazon RDS# Amazon RDS +# Amazon RDS ## Limitations -- No `SUPER` privileges. +- No `SUPER` privileges. - `gh-ost` runs should be setup use [`--assume-rbr`][assume_rbr_docs] and use `binlog_format=ROW`. - Aurora does not allow editing of the `read_only` parameter. While it is defined as `{TrueIfReplica}`, the parameter is non-modifiable field. @@ -10,7 +10,7 @@ #### Replication -In Aurora replication, you have separate reader and writer endpoints however because the cluster shares the underlying storage layer, `gh-ost` will detect it is running on the master. This becomes an issue when you wish to use [migrate/test on replica][migrate_test_on_replica_docs] because you won't be able to use a single cluster in the same way you would with MySQL RDS. +In Aurora replication, you have separate reader and writer endpoints however because the cluster shares the underlying storage layer, `gh-ost` will detect it is running on the master. This becomes an issue when you wish to use [migrate/test on replica][migrate_test_on_replica_docs] because you won't be able to use a single cluster in the same way you would with MySQL RDS. To work around this, you can follow along the [AWS replication between clusters documentation][aws_replication_docs] for Aurora with one small caveat. For the "Create a Snapshot of Your Replication Master" step, the binlog position is not available in the AWS console. You will need to issue the SQL query `SHOW SLAVE STATUS` or `aws rds describe-events` API call to get the correct position. @@ -24,6 +24,7 @@ If you use `pt-table-checksum` as a part of your data integrity checks, you migh This tool requires binlog_format=STATEMENT, but the current binlog_format is set to ROW and an error occurred while attempting to change it. If running MySQL 5.1.29 or newer, setting binlog_format requires the SUPER privilege. You will need to manually set binlog_format to 'STATEMENT' before running this tool. ``` +[assume_rbr_docs]: https://github.com/github/gh-ost/blob/master/doc/command-line-flags.md#assume-rbr [migrate_test_on_replica_docs]: https://github.com/github/gh-ost/blob/master/doc/cheatsheet.md#c-migratetest-on-replica [aws_replication_docs]: http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Aurora.Overview.Replication.MySQLReplication.html -[percona_toolkit_patch]: https://github.com/jacobbednarz/percona-toolkit/commit/0271ba6a094da446a5e5bb8d99b5c26f1777f2b9 \ No newline at end of file +[percona_toolkit_patch]: https://github.com/jacobbednarz/percona-toolkit/commit/0271ba6a094da446a5e5bb8d99b5c26f1777f2b9 From bf864e0e0c534494c4ff99de0abb917fd9dec9bf Mon Sep 17 00:00:00 2001 From: Jacob Bednarz Date: Fri, 24 Mar 2017 14:00:26 +1100 Subject: [PATCH 06/11] Add note on binlogs requiring backup > 1d --- doc/rds.md | 1 + 1 file changed, 1 insertion(+) diff --git a/doc/rds.md b/doc/rds.md index baec8fb..928fb83 100644 --- a/doc/rds.md +++ b/doc/rds.md @@ -5,6 +5,7 @@ - No `SUPER` privileges. - `gh-ost` runs should be setup use [`--assume-rbr`][assume_rbr_docs] and use `binlog_format=ROW`. - Aurora does not allow editing of the `read_only` parameter. While it is defined as `{TrueIfReplica}`, the parameter is non-modifiable field. +- In order to have binlogs enabled, the backup window must be set to greater than 1 day. ## Aurora From 1aaf47ec6a09f2f653a0c08c0a3ec6779125df33 Mon Sep 17 00:00:00 2001 From: Jacob Bednarz Date: Tue, 28 Mar 2017 12:02:39 +1100 Subject: [PATCH 07/11] Add note that `gh-ost` _does_ work with RDS --- doc/rds.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/doc/rds.md b/doc/rds.md index 928fb83..130a7ed 100644 --- a/doc/rds.md +++ b/doc/rds.md @@ -1,3 +1,5 @@ +`gh-ost` has been updated to work with Amazon RDS however due to GitHub not relying using AWS for databases, this documentation is community driven so if you find a bug please [open an issue][new_issue]! + # Amazon RDS ## Limitations @@ -25,6 +27,7 @@ If you use `pt-table-checksum` as a part of your data integrity checks, you migh This tool requires binlog_format=STATEMENT, but the current binlog_format is set to ROW and an error occurred while attempting to change it. If running MySQL 5.1.29 or newer, setting binlog_format requires the SUPER privilege. You will need to manually set binlog_format to 'STATEMENT' before running this tool. ``` +[new_issue]: https://github.com/github/gh-ost/issues/new [assume_rbr_docs]: https://github.com/github/gh-ost/blob/master/doc/command-line-flags.md#assume-rbr [migrate_test_on_replica_docs]: https://github.com/github/gh-ost/blob/master/doc/cheatsheet.md#c-migratetest-on-replica [aws_replication_docs]: http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Aurora.Overview.Replication.MySQLReplication.html From 55d8b3a188015254514dc6b84b0316d7a39efb38 Mon Sep 17 00:00:00 2001 From: Jacob Bednarz Date: Tue, 28 Mar 2017 17:03:32 +1100 Subject: [PATCH 08/11] Add preflight checklist for aurora docs --- doc/rds.md | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/doc/rds.md b/doc/rds.md index 130a7ed..3aee02b 100644 --- a/doc/rds.md +++ b/doc/rds.md @@ -7,7 +7,6 @@ - No `SUPER` privileges. - `gh-ost` runs should be setup use [`--assume-rbr`][assume_rbr_docs] and use `binlog_format=ROW`. - Aurora does not allow editing of the `read_only` parameter. While it is defined as `{TrueIfReplica}`, the parameter is non-modifiable field. -- In order to have binlogs enabled, the backup window must be set to greater than 1 day. ## Aurora @@ -27,6 +26,15 @@ If you use `pt-table-checksum` as a part of your data integrity checks, you migh This tool requires binlog_format=STATEMENT, but the current binlog_format is set to ROW and an error occurred while attempting to change it. If running MySQL 5.1.29 or newer, setting binlog_format requires the SUPER privilege. You will need to manually set binlog_format to 'STATEMENT' before running this tool. ``` +#### Preflight checklist + +Before trying to run any `gh-ost` migrations you will want to confirm the following: + +- [ ] You have a secondary cluster available that will act as a replica. Rule of thumb here has been a 1 instance per cluster to mimic MySQL-style replication as opposed to Aurora style. +- [ ] The database instance parameters and database cluster parameters are consistent between your master and replicas +- [ ] Executing `SHOW SLAVE STATUS\G` on your replica cluster displays the correct master host, binlog position, etc. +- [ ] Database backup retention is greater than 1 day to enable binlogs + [new_issue]: https://github.com/github/gh-ost/issues/new [assume_rbr_docs]: https://github.com/github/gh-ost/blob/master/doc/command-line-flags.md#assume-rbr [migrate_test_on_replica_docs]: https://github.com/github/gh-ost/blob/master/doc/cheatsheet.md#c-migratetest-on-replica From dfb9f888af3b8a2009cc77f16bbfae14fca48750 Mon Sep 17 00:00:00 2001 From: Jacob Bednarz Date: Mon, 3 Apr 2017 15:52:57 +1000 Subject: [PATCH 09/11] Add note for using hooks for stopping and starting RDS replication --- doc/rds.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/doc/rds.md b/doc/rds.md index 3aee02b..540284e 100644 --- a/doc/rds.md +++ b/doc/rds.md @@ -32,11 +32,14 @@ Before trying to run any `gh-ost` migrations you will want to confirm the follow - [ ] You have a secondary cluster available that will act as a replica. Rule of thumb here has been a 1 instance per cluster to mimic MySQL-style replication as opposed to Aurora style. - [ ] The database instance parameters and database cluster parameters are consistent between your master and replicas -- [ ] Executing `SHOW SLAVE STATUS\G` on your replica cluster displays the correct master host, binlog position, etc. +- [ ] Executing `SHOW SLAVE STATUS\G` on your replica cluster displays the correct master host, binlog position, etc. - [ ] Database backup retention is greater than 1 day to enable binlogs +- [ ] You have setup [`hooks`][ghost_hooks] to issue RDS procedures for stopping and starting replication. (see [github/gh-ost#163][ghost_rds_issue_tracking] for examples) [new_issue]: https://github.com/github/gh-ost/issues/new [assume_rbr_docs]: https://github.com/github/gh-ost/blob/master/doc/command-line-flags.md#assume-rbr [migrate_test_on_replica_docs]: https://github.com/github/gh-ost/blob/master/doc/cheatsheet.md#c-migratetest-on-replica [aws_replication_docs]: http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Aurora.Overview.Replication.MySQLReplication.html [percona_toolkit_patch]: https://github.com/jacobbednarz/percona-toolkit/commit/0271ba6a094da446a5e5bb8d99b5c26f1777f2b9 +[ghost_hooks]: https://envato.slack.com/archives/C08KD4AQJ/p1491197511461443 +[ghost_rds_issue_tracking]: https://github.com/github/gh-ost/issues/163 From 2f48e89ea0c6cf856d7c25c57e311d021b4ecbf2 Mon Sep 17 00:00:00 2001 From: Jacob Bednarz Date: Mon, 3 Apr 2017 15:59:11 +1000 Subject: [PATCH 10/11] This isn't a ghost hook URL, use the _real_ one --- doc/rds.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/rds.md b/doc/rds.md index 540284e..889d480 100644 --- a/doc/rds.md +++ b/doc/rds.md @@ -41,5 +41,5 @@ Before trying to run any `gh-ost` migrations you will want to confirm the follow [migrate_test_on_replica_docs]: https://github.com/github/gh-ost/blob/master/doc/cheatsheet.md#c-migratetest-on-replica [aws_replication_docs]: http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Aurora.Overview.Replication.MySQLReplication.html [percona_toolkit_patch]: https://github.com/jacobbednarz/percona-toolkit/commit/0271ba6a094da446a5e5bb8d99b5c26f1777f2b9 -[ghost_hooks]: https://envato.slack.com/archives/C08KD4AQJ/p1491197511461443 +[ghost_hooks]: https://github.com/github/gh-ost/blob/master/doc/hooks.md [ghost_rds_issue_tracking]: https://github.com/github/gh-ost/issues/163 From 098994452881f88811c3b3a1629d3b91ec3ab414 Mon Sep 17 00:00:00 2001 From: Jacob Bednarz Date: Mon, 10 Apr 2017 13:58:35 +1000 Subject: [PATCH 11/11] Add RDS link in README usage --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 04c032a..afc4cf6 100644 --- a/README.md +++ b/README.md @@ -64,6 +64,7 @@ Also see: - [what if?](doc/what-if.md) - [the fine print](doc/the-fine-print.md) - [Community questions](https://github.com/github/gh-ost/issues?q=label%3Aquestion) +- [Using `gh-ost` on AWS RDS](doc/rds.md) ## What's in a name?