Commit Graph

232 Commits

Author SHA1 Message Date
Morgan Tocker
59c1a24774 remove useless func per review 2022-11-15 15:40:21 -07:00
Morgan Tocker
74fb8e80b2
Update go/logic/migrator.go
Co-authored-by: dm-2 <45519614+dm-2@users.noreply.github.com>
2022-11-15 12:05:27 -07:00
Morgan Tocker
3f3a10a213 Address PR feedback 2022-11-14 14:48:49 -07:00
Tim Vaillancourt
cfd83728ed
Merge branch 'master' into attempt-instant-ddl 2022-11-14 22:09:55 +01:00
Morgan Tocker
75a346be93 Add tests, incorporate feedback 2022-11-14 11:47:30 -07:00
Morgan Tocker
05f32ebf29 minor cleanup 2022-11-10 09:30:13 -07:00
Morgan Tocker
1be6a4c082 Attempt instant DDL if supported 2022-11-09 20:11:49 -07:00
Nicholas Calugar
515aa72d3d Print status to migration context logger 2022-10-27 11:42:12 -07:00
Tim Vaillancourt
9b3fa793ac
Enable more golangci-lint linters (#1181) 2022-10-21 18:02:06 +02:00
Tim Vaillancourt
df4cf7b38e
Add basic test for hooks (#1179) 2022-10-21 17:47:39 +02:00
Tim Vaillancourt
1df37c207f
Add basic tests to migrator (#1168) 2022-09-06 15:48:48 +02:00
Tim Vaillancourt
ffe54f48ed
Enable more golang-ci linters (#1149) 2022-07-18 18:37:18 +02:00
Tim Vaillancourt
54db4174b3 Consolidate no-op states 2022-07-07 17:40:01 +02:00
Tim Vaillancourt
246800e053 Fix lint 2022-07-07 17:38:41 +02:00
Tim Vaillancourt
d6d1d3b436
Merge branch 'master' into changelog-migrated-state 2022-07-07 17:35:59 +02:00
Tim Vaillancourt
eead61cb6d cleanup v4 2022-07-07 05:05:37 +02:00
Tim Vaillancourt
0303031fc2 cleanup v2 2022-07-07 03:22:53 +02:00
Tim Vaillancourt
500fbefdb9 golang-ci: fix staticcheck linter warnings 2022-07-07 03:12:44 +02:00
Andrew Mason
b751499091
Cancel any row count queries before attempting to cut over (#846)
* Cancel any row count queries before attempting to cut over

Closes #830. Switches from using `QueryRow` to `QueryRowContext`, and
stores a context.CancelFunc in the migration context, which is called to
halt any running row count query before beginning the cut over.

* Make it threadsafe

* Kill the count query on the database side as well

* Explicitly grab a connection to run the count, store its connection id
* When the query context is canceled, run a `KILL QUERY ?` on that connection id

* Rewrite these to use the threadsafe functions, stop exporting the cancel func

* Update logger

* Update logger

Co-authored-by: Tim Vaillancourt <timvaillancourt@github.com>
Co-authored-by: Tim Vaillancourt <tim@timvaillancourt.com>
Co-authored-by: dm-2 <45519614+dm-2@users.noreply.github.com>
2022-07-07 00:23:23 +02:00
Tim Vaillancourt
0918bab29b
Add context/timeout to HTTP throttle check (#1131)
* Add context/timeout to HTTP throttle check

* Dont run `.GetThrottleHTTPInterval()` on every loop

* Update help message

* Var rename

* 2022

* Add timeout flag

* Add unix/tcp server commands, use ParseInt() for string->int64

* Var rename

* Re-check http timeout on every loop iteration

* Remove stale comment

* Make throttle interval idempotent

* var rename

* Usage grammar

* Make http timeout idempotent too

* Parse time.Duration once

* Move timeout to NewThrottler

* Help update

* Set User-Agent header

* Re-add newline

Co-authored-by: dm-2 <45519614+dm-2@users.noreply.github.com>
2022-07-06 23:56:07 +02:00
Tim Vaillancourt
0b066c16a5
Use switch statements for readability, simplify .NewGoMySQLReader() (#1135)
* Use `switch` statements for readability

* Simplify initBinlogReader()
2022-07-06 23:45:26 +02:00
dm-2
b7f7e2c0b0
Merge branch 'master' into fix-issue-1117 2022-07-06 16:58:51 +01:00
Tim Vaillancourt
38e86e5791
Merge branch 'master' into fix-two-phase-commit-lost-data_v2 2022-06-25 00:37:12 +02:00
shaohk
87970f6312 fix(lost data in mysql two-phase commit): lost data in mysql two-phase commit 2022-06-24 11:38:05 +08:00
Tim Vaillancourt
de339dd042 Fix integer divide by zero panic in migrator 2022-06-23 03:37:18 +02:00
Tim Vaillancourt
ec199f1185 revert out of scope change 2022-06-05 00:06:41 +02:00
Tim Vaillancourt
af99b247f9 Add Migrated changelog event 2022-06-04 23:59:00 +02:00
Tim Vaillancourt
ed46138c06
Add golangci-lint CI action, fix gosimple, govet + unused lint errors (#1127)
* Add `golangci-lint`, fix `gosimple`, `govet` and `unused` linter complaints

* Go 1.16

* Update copyright dates
2022-05-31 21:23:39 +02:00
Tyler Knodell
fbb53f4b04 Add flag to customize the interval which the onStatus hook is called 2022-02-01 08:54:55 -07:00
Shlomi Noach
f19f101610
hooks: reporting GH_OST_ETA_SECONDS. ETA as part of migration context (#936)
* v1.1.0

* WIP: copying AUTO_INCREMENT value to ghost table
Initial commit: towards setting up a test suite

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* greping for 'expect_table_structure' content

* Adding simple test for 'expect_table_structure' scenario

* adding tests for AUTO_INCREMENT value after row deletes. Should initially fail

* clear event beforehand

* parsing AUTO_INCREMENT from alter query, reading AUTO_INCREMENT from original table, applying AUTO_INCREMENT value onto ghost table if applicable and user has not specified AUTO_INCREMENT in alter statement

* support GetUint64

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* minor update to test

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* adding test for user defined AUTO_INCREMENT statement

* Generated column as part of UNIQUE (or PRIMARY) KEY

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* skip analysis of generated column data type in unique key

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* All MySQL DBs limited to max 3 concurrent/idle connections

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* hooks: reporting GH_OST_ETA_SECONDS. ETA stored as part of migration context

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* GH_OST_ETA_NANOSECONDS

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* N/A denoted by negative value

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* ETAUnknown constant

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
2021-05-31 14:15:51 +02:00
Tim Vaillancourt
c71dbf9ef3
Copy auto increment (#967)
* v1.1.0

* WIP: copying AUTO_INCREMENT value to ghost table
Initial commit: towards setting up a test suite

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* greping for 'expect_table_structure' content

* Adding simple test for 'expect_table_structure' scenario

* adding tests for AUTO_INCREMENT value after row deletes. Should initially fail

* clear event beforehand

* parsing AUTO_INCREMENT from alter query, reading AUTO_INCREMENT from original table, applying AUTO_INCREMENT value onto ghost table if applicable and user has not specified AUTO_INCREMENT in alter statement

* support GetUint64

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* minor update to test

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* adding test for user defined AUTO_INCREMENT statement

Co-authored-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
2021-05-14 15:32:56 +02:00
Cathal Coffey
3135a25c1f HeartbeatLag must be < than --max-lag-millis and --cut-over-lock-timeout-seconds to continue 2021-02-07 14:31:40 +00:00
Cathal Coffey
503b7b0d6c Consolidate the two sleepWhileTrue loops 2021-02-07 13:52:59 +00:00
Cathal Coffey
d5c2414893 Move 'heartbeat is too high' back to Debug logs again 2021-02-05 10:12:54 +00:00
Cathal Coffey
48ce0873de Store lastHeartbeatOnChangelogTime instead of CurrentHeartbeatLag 2021-02-03 10:03:45 +00:00
Cathal Coffey
4efd156759 Move 'heartbeat is too high' to Debug logs 2021-02-02 09:17:31 +00:00
Cathal Coffey
8a26c9ebf4 Don't cut-over until it is safe to do so 2021-02-01 10:14:05 +00:00
Cathal Coffey
a4218cd6f4 Progress should print HeartbeatLag 2021-01-31 18:27:34 +00:00
Cathal Coffey
8aee288fd7 Handle onChangelogHeartbeatEvent and update CurrentHeartbeatLag 2021-01-31 18:24:26 +00:00
Cathal Coffey
7207bc146a Make it easier to handle different onChangelogEvents 2021-01-31 18:23:09 +00:00
Tim Vaillancourt
8c4cd10d79
Merge branch 'master' into ipv6 2020-10-22 11:39:46 +02:00
Tim Vaillancourt
0e2d33ad86 Merge in https://github.com/github/gh-ost/pull/755 2020-10-20 16:08:49 +02:00
Shlomi Noach
f482356a94
Merge branch 'master' into parse-alter-statement 2020-07-22 12:36:59 +03:00
Shlomi Noach
6c7b4736e1 Support a complete ALTER TABLE statement in --alter 2020-07-22 12:33:02 +03:00
Shlomi Noach
a36d6bee94
Merge branch 'logging-interface-contrib' into master 2019-12-11 07:03:37 +02:00
Shlomi Noach
d0ce7c014e truncate lag digits 2019-10-23 10:18:56 +03:00
Abeyu M
f4676bf463 implement a logging interface 2019-10-07 11:10:36 -04:00
Shlomi Noach
21e7ec6b7c lag: %+vs notation, e.g. 1s 2019-10-07 07:06:11 +03:00
Shlomi Noach
271c7274a3 refactor progressPct into migrationContext 2019-10-07 06:59:56 +03:00
Shlomi Noach
7869889988 context, status and hooks: progressPct and CurrentLag 2019-10-06 17:08:35 +03:00