From e017ec18e466dffb40b30c19c99b418e6345a033 Mon Sep 17 00:00:00 2001 From: Jess Breckenridge Date: Wed, 3 May 2017 14:25:31 -0600 Subject: [PATCH 1/9] - Initial documentation on contributing to gh-ost. - It is a bit sparse currently, but will give beginners an idea how on to setup the environment and run tests. - A good starting point for further PR's. --- doc/codingghost.md | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) create mode 100644 doc/codingghost.md diff --git a/doc/codingghost.md b/doc/codingghost.md new file mode 100644 index 0000000..9828dac --- /dev/null +++ b/doc/codingghost.md @@ -0,0 +1,23 @@ +# Getting started with gh-ost development. + +## Overview + +Getting started with gh-ost development is simple! + +- First clone the repository. +- From inside of the repository run `script/cibuild` +- This will bootstrap the environment if needed, format the code, build the code, and then run the unit test. + +## CI build workflow + +`script/cibuild` performs the following actions: + +- It runs `script/bootstrap` +- `script/bootstrap` run `script/ensure-go-installed` +- `script/ensure-go-installed` installs go locally if (go is not installed) || (go is not version 1.7). It also will not install go if it is already installed locally. +- `script/build` builds the binary and places in in `bin/` + +## Notes: + +Currently, `script/ensure-go-installed` will install `go` for Mac OS X and Linux. We welcome PR's to add other platforms. + From 95c9547d541e5926894418771cd4f3479b68e38e Mon Sep 17 00:00:00 2001 From: Jess Breckenridge Date: Wed, 3 May 2017 14:31:45 -0600 Subject: [PATCH 2/9] - Initial documentation on getting started developing for gh-ost. --- doc/{codingghost.md => coding-ghost.md} | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) rename doc/{codingghost.md => coding-ghost.md} (70%) diff --git a/doc/codingghost.md b/doc/coding-ghost.md similarity index 70% rename from doc/codingghost.md rename to doc/coding-ghost.md index 9828dac..0023b9f 100644 --- a/doc/codingghost.md +++ b/doc/coding-ghost.md @@ -13,11 +13,10 @@ Getting started with gh-ost development is simple! `script/cibuild` performs the following actions: - It runs `script/bootstrap` -- `script/bootstrap` run `script/ensure-go-installed` -- `script/ensure-go-installed` installs go locally if (go is not installed) || (go is not version 1.7). It also will not install go if it is already installed locally. -- `script/build` builds the binary and places in in `bin/` +- `script/bootstrap` runs `script/ensure-go-installed` +- `script/ensure-go-installed` installs go locally if (go is not installed) || (go is not version 1.7). It will not install go if it is already installed locally and at the correct version. +- `script/build` builds the `gh-ost` binary and places in in `bin/` ## Notes: Currently, `script/ensure-go-installed` will install `go` for Mac OS X and Linux. We welcome PR's to add other platforms. - From 13491a0d0bf1f26b73c8cdcff73902b3024d42d9 Mon Sep 17 00:00:00 2001 From: Jess Breckenridge Date: Wed, 3 May 2017 14:35:31 -0600 Subject: [PATCH 3/9] - Adding link to `coding-ghost.md` documentation. --- README.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/README.md b/README.md index afc4cf6..086fa06 100644 --- a/README.md +++ b/README.md @@ -84,6 +84,8 @@ But then a rare genetic mutation happened, and the `c` transformed into `t`. And We develop `gh-ost` at GitHub and for the community. We may have different priorities than others. From time to time we may suggest a contribution that is not on our immediate roadmap but which may appeal to others. +Please see [Coding gh-ost](https://github.com/github/gh-ost/blob/develdocs/doc/command-line-flags.md) for a guide to getting started developing with gh-ost. + ## Download/binaries/source `gh-ost` is now GA and stable. From 7df2e0d433d5943a0e56a9574e6791adb6dd8319 Mon Sep 17 00:00:00 2001 From: Jess Breckenridge Date: Thu, 4 May 2017 12:51:00 -0600 Subject: [PATCH 4/9] - Updating PR template to reflect current build workflow --- .github/PULL_REQUEST_TEMPLATE.md | 4 +--- doc/coding-ghost.md | 9 +++------ 2 files changed, 4 insertions(+), 9 deletions(-) diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md index 4dc48fd..8301e70 100644 --- a/.github/PULL_REQUEST_TEMPLATE.md +++ b/.github/PULL_REQUEST_TEMPLATE.md @@ -16,6 +16,4 @@ This PR [briefly explain what is does] > In case this PR introduced Go code changes: - [ ] contributed code is using same conventions as original code -- [ ] code is formatted via `gofmt` (please avoid `goimports`) -- [ ] code is built via `./build.sh` -- [ ] code is tested via `./test.sh` +- [ ] `script/cibuild` returns with no formatting errors, build errors or unit test errors. diff --git a/doc/coding-ghost.md b/doc/coding-ghost.md index 0023b9f..ee26f0c 100644 --- a/doc/coding-ghost.md +++ b/doc/coding-ghost.md @@ -4,18 +4,15 @@ Getting started with gh-ost development is simple! -- First clone the repository. +- First obtain the repository with `git clone` or `go get`. - From inside of the repository run `script/cibuild` - This will bootstrap the environment if needed, format the code, build the code, and then run the unit test. ## CI build workflow -`script/cibuild` performs the following actions: +`script/cibuild` performs the following actions will bootstrap the environment to build `gh-ost` correctly, build, perform syntax checks and run unit tests. -- It runs `script/bootstrap` -- `script/bootstrap` runs `script/ensure-go-installed` -- `script/ensure-go-installed` installs go locally if (go is not installed) || (go is not version 1.7). It will not install go if it is already installed locally and at the correct version. -- `script/build` builds the `gh-ost` binary and places in in `bin/` +If additional steps are needed, please add them into this workflow so that the workflow remains simple. ## Notes: From b79185ab6471a13f11e87540e0580c6da27af897 Mon Sep 17 00:00:00 2001 From: Jess Breckenridge Date: Thu, 4 May 2017 12:58:22 -0600 Subject: [PATCH 5/9] - Bad copy and paste. Coding gh-ost in README.md actually pointed to command-line-flags.md. --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 086fa06..8de361b 100644 --- a/README.md +++ b/README.md @@ -84,7 +84,7 @@ But then a rare genetic mutation happened, and the `c` transformed into `t`. And We develop `gh-ost` at GitHub and for the community. We may have different priorities than others. From time to time we may suggest a contribution that is not on our immediate roadmap but which may appeal to others. -Please see [Coding gh-ost](https://github.com/github/gh-ost/blob/develdocs/doc/command-line-flags.md) for a guide to getting started developing with gh-ost. +Please see [Coding gh-ost](https://github.com/github/gh-ost/blob/develdocs/doc/coding-ghost.md) for a guide to getting started developing with gh-ost. ## Download/binaries/source From 3955a6d67f80404d1f2611b82023ba4daeda423c Mon Sep 17 00:00:00 2001 From: Shlomi Noach Date: Wed, 24 May 2017 08:32:13 +0300 Subject: [PATCH 6/9] hibernate on critical-load --- go/base/context.go | 2 ++ go/cmd/gh-ost/main.go | 1 + go/logic/applier.go | 3 +++ go/logic/throttler.go | 30 ++++++++++++++++++++++++++++++ 4 files changed, 36 insertions(+) diff --git a/go/base/context.go b/go/base/context.go index c300df1..357afab 100644 --- a/go/base/context.go +++ b/go/base/context.go @@ -105,9 +105,11 @@ type MigrationContext struct { throttleQuery string throttleHTTP string ThrottleCommandedByUser int64 + HibernateUntil int64 maxLoad LoadMap criticalLoad LoadMap CriticalLoadIntervalMilliseconds int64 + CriticalLoadHibernateSeconds int64 PostponeCutOverFlagFile string CutOverLockTimeoutSeconds int64 ForceNamedCutOverCommand bool diff --git a/go/cmd/gh-ost/main.go b/go/cmd/gh-ost/main.go index f27e12b..a4f4f3e 100644 --- a/go/cmd/gh-ost/main.go +++ b/go/cmd/gh-ost/main.go @@ -112,6 +112,7 @@ func main() { maxLoad := flag.String("max-load", "", "Comma delimited status-name=threshold. e.g: 'Threads_running=100,Threads_connected=500'. When status exceeds threshold, app throttles writes") criticalLoad := flag.String("critical-load", "", "Comma delimited status-name=threshold, same format as --max-load. When status exceeds threshold, app panics and quits") flag.Int64Var(&migrationContext.CriticalLoadIntervalMilliseconds, "critical-load-interval-millis", 0, "When 0, migration immediately bails out upon meeting critical-load. When non-zero, a second check is done after given interval, and migration only bails out if 2nd check still meets critical load") + flag.Int64Var(&migrationContext.CriticalLoadHibernateSeconds, "critical-load-hibernate-seconds", 0, "When nonzero, critical-load does not panic and bail out; instead, gh-ost goes into hibernate for the specified duration. It will not read/write anything to from/to any server") quiet := flag.Bool("quiet", false, "quiet") verbose := flag.Bool("verbose", false, "verbose") debug := flag.Bool("debug", false, "debug mode (very verbose)") diff --git a/go/logic/applier.go b/go/logic/applier.go index 4e3f783..b167de8 100644 --- a/go/logic/applier.go +++ b/go/logic/applier.go @@ -293,6 +293,9 @@ func (this *Applier) WriteChangelogState(value string) (string, error) { func (this *Applier) InitiateHeartbeat() { var numSuccessiveFailures int64 injectHeartbeat := func() error { + if atomic.LoadInt64(&this.migrationContext.HibernateUntil) > 0 { + return nil + } if _, err := this.WriteChangelog("heartbeat", time.Now().Format(time.RFC3339Nano)); err != nil { numSuccessiveFailures++ if numSuccessiveFailures > this.migrationContext.MaxRetries() { diff --git a/go/logic/throttler.go b/go/logic/throttler.go index 1c2c62a..808fab4 100644 --- a/go/logic/throttler.go +++ b/go/logic/throttler.go @@ -38,6 +38,10 @@ func NewThrottler(applier *Applier, inspector *Inspector) *Throttler { // It merely observes the metrics collected by other components, it does not issue // its own metric collection. func (this *Throttler) shouldThrottle() (result bool, reason string, reasonHint base.ThrottleReasonHint) { + if hibernateUntil := atomic.LoadInt64(&this.migrationContext.HibernateUntil); hibernateUntil > 0 { + hibernateUntilTime := time.Unix(0, hibernateUntil) + return true, fmt.Sprintf("critical-load-hibernate until %+v", hibernateUntilTime), base.NoThrottleReasonHint + } generalCheckResult := this.migrationContext.GetThrottleGeneralCheckResult() if generalCheckResult.ShouldThrottle { return generalCheckResult.ShouldThrottle, generalCheckResult.Reason, generalCheckResult.ReasonHint @@ -96,6 +100,9 @@ func (this *Throttler) collectReplicationLag(firstThrottlingCollected chan<- boo if atomic.LoadInt64(&this.migrationContext.CleanupImminentFlag) > 0 { return nil } + if atomic.LoadInt64(&this.migrationContext.HibernateUntil) > 0 { + return nil + } if this.migrationContext.TestOnReplica || this.migrationContext.MigrateOnReplica { // when running on replica, the heartbeat injection is also done on the replica. @@ -128,6 +135,10 @@ func (this *Throttler) collectReplicationLag(firstThrottlingCollected chan<- boo // collectControlReplicasLag polls all the control replicas to get maximum lag value func (this *Throttler) collectControlReplicasLag() { + if atomic.LoadInt64(&this.migrationContext.HibernateUntil) > 0 { + return + } + replicationLagQuery := fmt.Sprintf(` select value from %s.%s where hint = 'heartbeat' and id <= 255 `, @@ -222,6 +233,9 @@ func (this *Throttler) criticalLoadIsMet() (met bool, variableName string, value // collectReplicationLag reads the latest changelog heartbeat value func (this *Throttler) collectThrottleHTTPStatus(firstThrottlingCollected chan<- bool) { collectFunc := func() (sleep bool, err error) { + if atomic.LoadInt64(&this.migrationContext.HibernateUntil) > 0 { + return true, nil + } url := this.migrationContext.GetThrottleHTTP() if url == "" { return true, nil @@ -247,6 +261,9 @@ func (this *Throttler) collectThrottleHTTPStatus(firstThrottlingCollected chan<- // collectGeneralThrottleMetrics reads the once-per-sec metrics, and stores them onto this.migrationContext func (this *Throttler) collectGeneralThrottleMetrics() error { + if atomic.LoadInt64(&this.migrationContext.HibernateUntil) > 0 { + return nil + } setThrottle := func(throttle bool, reason string, reasonHint base.ThrottleReasonHint) error { this.migrationContext.SetThrottleGeneralCheckResult(base.NewThrottleCheckResult(throttle, reason, reasonHint)) @@ -264,6 +281,19 @@ func (this *Throttler) collectGeneralThrottleMetrics() error { if err != nil { return setThrottle(true, fmt.Sprintf("%s %s", variableName, err), base.NoThrottleReasonHint) } + + if this.migrationContext.CriticalLoadHibernateSeconds > 0 { + hibernateDuration := time.Duration(this.migrationContext.CriticalLoadHibernateSeconds) * time.Second + hibernateUntilTime := time.Now().Add(hibernateDuration) + atomic.StoreInt64(&this.migrationContext.HibernateUntil, hibernateUntilTime.UnixNano()) + log.Errorf("critical-load met. Will hibernate for the duration of %+v, until %+v", hibernateDuration, hibernateUntilTime) + go func() { + time.Sleep(hibernateDuration) + atomic.StoreInt64(&this.migrationContext.HibernateUntil, 0) + }() + return nil + } + if criticalLoadMet && this.migrationContext.CriticalLoadIntervalMilliseconds == 0 { this.migrationContext.PanicAbort <- fmt.Errorf("critical-load met: %s=%d, >=%d", variableName, value, threshold) } From ad47f7c1477780020631627514d3ef168e8f71ff Mon Sep 17 00:00:00 2001 From: Shlomi Noach Date: Wed, 24 May 2017 10:42:47 +0300 Subject: [PATCH 7/9] throttling just prior to leaving hibernation, so as to allow re-throttle checks to apply --- go/base/context.go | 5 +++-- go/logic/throttler.go | 1 + 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/go/base/context.go b/go/base/context.go index 357afab..d82b22e 100644 --- a/go/base/context.go +++ b/go/base/context.go @@ -40,8 +40,9 @@ const ( type ThrottleReasonHint string const ( - NoThrottleReasonHint ThrottleReasonHint = "NoThrottleReasonHint" - UserCommandThrottleReasonHint = "UserCommandThrottleReasonHint" + NoThrottleReasonHint ThrottleReasonHint = "NoThrottleReasonHint" + UserCommandThrottleReasonHint = "UserCommandThrottleReasonHint" + LeavingHibernationThrottleReasonHint = "LeavingHibernationThrottleReasonHint" ) const ( diff --git a/go/logic/throttler.go b/go/logic/throttler.go index 808fab4..8f21c0d 100644 --- a/go/logic/throttler.go +++ b/go/logic/throttler.go @@ -289,6 +289,7 @@ func (this *Throttler) collectGeneralThrottleMetrics() error { log.Errorf("critical-load met. Will hibernate for the duration of %+v, until %+v", hibernateDuration, hibernateUntilTime) go func() { time.Sleep(hibernateDuration) + this.migrationContext.SetThrottleGeneralCheckResult(base.NewThrottleCheckResult(true, "leaving hibernation", base.LeavingHibernationThrottleReasonHint)) atomic.StoreInt64(&this.migrationContext.HibernateUntil, 0) }() return nil From 8da0f60582770b13e024efa65ec4f823c1e595ca Mon Sep 17 00:00:00 2001 From: Shlomi Noach Date: Wed, 24 May 2017 10:53:00 +0300 Subject: [PATCH 8/9] fixed critical-load check for hibernation --- go/logic/throttler.go | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/go/logic/throttler.go b/go/logic/throttler.go index 8f21c0d..ecc1f2b 100644 --- a/go/logic/throttler.go +++ b/go/logic/throttler.go @@ -282,11 +282,11 @@ func (this *Throttler) collectGeneralThrottleMetrics() error { return setThrottle(true, fmt.Sprintf("%s %s", variableName, err), base.NoThrottleReasonHint) } - if this.migrationContext.CriticalLoadHibernateSeconds > 0 { + if criticalLoadMet && this.migrationContext.CriticalLoadHibernateSeconds > 0 { hibernateDuration := time.Duration(this.migrationContext.CriticalLoadHibernateSeconds) * time.Second hibernateUntilTime := time.Now().Add(hibernateDuration) atomic.StoreInt64(&this.migrationContext.HibernateUntil, hibernateUntilTime.UnixNano()) - log.Errorf("critical-load met. Will hibernate for the duration of %+v, until %+v", hibernateDuration, hibernateUntilTime) + log.Errorf("critical-load met: %s=%d, >=%d. Will hibernate for the duration of %+v, until %+v", variableName, value, threshold, hibernateDuration, hibernateUntilTime) go func() { time.Sleep(hibernateDuration) this.migrationContext.SetThrottleGeneralCheckResult(base.NewThrottleCheckResult(true, "leaving hibernation", base.LeavingHibernationThrottleReasonHint)) From 83c2c7dc230e6019697835b4251433fb06fd3571 Mon Sep 17 00:00:00 2001 From: Greg Roodt Date: Mon, 5 Jun 2017 15:59:24 +1000 Subject: [PATCH 9/9] Update requirements-and-limitations.md --- doc/requirements-and-limitations.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/requirements-and-limitations.md b/doc/requirements-and-limitations.md index 7d246a8..c961706 100644 --- a/doc/requirements-and-limitations.md +++ b/doc/requirements-and-limitations.md @@ -40,8 +40,8 @@ The `SUPER` privilege is required for `STOP SLAVE`, `START SLAVE` operations. Th - It is not allowed to migrate a table where another table exists with same name and different upper/lower case. - For example, you may not migrate `MyTable` if another table called `MYtable` exists in the same schema. -- Amazon RDS and Google Cloud SQL are currently not supported - - We began working towards removing this limitation. See tracking issue: https://github.com/github/gh-ost/issues/163 +- Amazon RDS works, but has it's own [limitations](rds.md). +- Google Cloud SQL is currently not supported - Multisource is not supported when migrating via replica. It _should_ work (but never tested) when connecting directly to master (`--allow-on-master`)