Commit Graph

437 Commits

Author SHA1 Message Date
Tim Vaillancourt
38e86e5791
Merge branch 'master' into fix-two-phase-commit-lost-data_v2 2022-06-25 00:37:12 +02:00
shaohk
87970f6312 fix(lost data in mysql two-phase commit): lost data in mysql two-phase commit 2022-06-24 11:38:05 +08:00
Tim Vaillancourt
ed46138c06
Add golangci-lint CI action, fix gosimple, govet + unused lint errors (#1127)
* Add `golangci-lint`, fix `gosimple`, `govet` and `unused` linter complaints

* Go 1.16

* Update copyright dates
2022-05-31 21:23:39 +02:00
Tim Vaillancourt
5ea949f31e Remove end-of-life MySQL 5.5.x 2022-02-25 00:02:27 +01:00
dm-2
260e662197
Merge branch 'master' into inspector-connectionConfig-instanceKey-string 2022-02-24 10:34:19 +00:00
Tim Vaillancourt
1aa74bfe60 Fix needless fmt.Sprintf call in go/logic/inspector.go 2022-02-24 01:30:29 +01:00
Tim Vaillancourt
f6db40e4cd Use .String() for logging connection-config InstanceKey 2022-02-23 23:28:03 +01:00
Tyler Knodell
fbb53f4b04 Add flag to customize the interval which the onStatus hook is called 2022-02-01 08:54:55 -07:00
Andrew Mason
6e1daf90ee
Check RowsAffected when applying DML events to get more accurate statistics (#844)
* Check RowsAffected when applying DML events to get more accurate statistics

Addresses #600.

When applying a DML event, check the RowsAffected on the `Result`
struct. Since all DML event queries are point queries, this will only
ever be 0 or 1. The applier then takes this value and multiplies by
the `rowsDelta` of the event, resulting in a properly-signed, accurate
row delta to use in the statistics.

If an error occurs here, log it, but do not surface this as an
actual error .. simply assume the DML affected a row and move on. It
will be inaccurate, but this is already the case.

* Fix import

* update wording to warning log message

Co-authored-by: Tim Vaillancourt <timvaillancourt@github.com>

Co-authored-by: Tim Vaillancourt <timvaillancourt@github.com>
2021-07-14 16:48:03 +02:00
Tim Vaillancourt
47d49c6b92
Add go mod (#935)
* Add a go.mod file

* run go mod vendor again

* Move to a well-supported ini file reader

* Remove GO111MODULE=off

* Use go 1.16

* Rename github.com/outbrain/golib -> github.com/openark/golib

* Remove *.go-e files

* Fix for `strconv.ParseInt: parsing "": invalid syntax` error

* Add test for '[osc]' section

Co-authored-by: Nate Wernimont <nate.wernimont@workiva.com>
2021-06-24 20:19:37 +02:00
Shlomi Noach
9bc508f068
Enum to varchar (#963)
* v1.1.0

* WIP: copying AUTO_INCREMENT value to ghost table
Initial commit: towards setting up a test suite

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* greping for 'expect_table_structure' content

* Adding simple test for 'expect_table_structure' scenario

* adding tests for AUTO_INCREMENT value after row deletes. Should initially fail

* clear event beforehand

* parsing AUTO_INCREMENT from alter query, reading AUTO_INCREMENT from original table, applying AUTO_INCREMENT value onto ghost table if applicable and user has not specified AUTO_INCREMENT in alter statement

* support GetUint64

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* minor update to test

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* adding test for user defined AUTO_INCREMENT statement

* Generated column as part of UNIQUE (or PRIMARY) KEY

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* skip analysis of generated column data type in unique key

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* All MySQL DBs limited to max 3 concurrent/idle connections

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* hooks: reporting GH_OST_ETA_SECONDS. ETA stored as part of migration context

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* GH_OST_ETA_NANOSECONDS

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* N/A denoted by negative value

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* ETAUnknown constant

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* Convering enum to varchar

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* test: not null

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* first attempt at setting enum-to-string right

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* fix insert query

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* store enum values, use when populating

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* apply EnumValues to mapped column

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* fix compilation error

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* gofmt

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
2021-06-10 17:17:49 +02:00
Shlomi Noach
f19f101610
hooks: reporting GH_OST_ETA_SECONDS. ETA as part of migration context (#936)
* v1.1.0

* WIP: copying AUTO_INCREMENT value to ghost table
Initial commit: towards setting up a test suite

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* greping for 'expect_table_structure' content

* Adding simple test for 'expect_table_structure' scenario

* adding tests for AUTO_INCREMENT value after row deletes. Should initially fail

* clear event beforehand

* parsing AUTO_INCREMENT from alter query, reading AUTO_INCREMENT from original table, applying AUTO_INCREMENT value onto ghost table if applicable and user has not specified AUTO_INCREMENT in alter statement

* support GetUint64

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* minor update to test

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* adding test for user defined AUTO_INCREMENT statement

* Generated column as part of UNIQUE (or PRIMARY) KEY

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* skip analysis of generated column data type in unique key

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* All MySQL DBs limited to max 3 concurrent/idle connections

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* hooks: reporting GH_OST_ETA_SECONDS. ETA stored as part of migration context

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* GH_OST_ETA_NANOSECONDS

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* N/A denoted by negative value

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* ETAUnknown constant

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
2021-05-31 14:15:51 +02:00
Shlomi Noach
c41823ecc9
All MySQL DBs limited to max 3 concurrent/idle connections #15 (#931)
* v1.1.0

* WIP: copying AUTO_INCREMENT value to ghost table
Initial commit: towards setting up a test suite

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* greping for 'expect_table_structure' content

* Adding simple test for 'expect_table_structure' scenario

* adding tests for AUTO_INCREMENT value after row deletes. Should initially fail

* clear event beforehand

* parsing AUTO_INCREMENT from alter query, reading AUTO_INCREMENT from original table, applying AUTO_INCREMENT value onto ghost table if applicable and user has not specified AUTO_INCREMENT in alter statement

* support GetUint64

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* minor update to test

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* adding test for user defined AUTO_INCREMENT statement

* Generated column as part of UNIQUE (or PRIMARY) KEY

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* skip analysis of generated column data type in unique key

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* All MySQL DBs limited to max 3 concurrent/idle connections

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
2021-05-27 20:00:58 +02:00
Shlomi Noach
36c669dd75
Generated column as part of UNIQUE (or PRIMARY) KEY (#919)
* v1.1.0

* WIP: copying AUTO_INCREMENT value to ghost table
Initial commit: towards setting up a test suite

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* greping for 'expect_table_structure' content

* Adding simple test for 'expect_table_structure' scenario

* adding tests for AUTO_INCREMENT value after row deletes. Should initially fail

* clear event beforehand

* parsing AUTO_INCREMENT from alter query, reading AUTO_INCREMENT from original table, applying AUTO_INCREMENT value onto ghost table if applicable and user has not specified AUTO_INCREMENT in alter statement

* support GetUint64

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* minor update to test

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* adding test for user defined AUTO_INCREMENT statement

* Generated column as part of UNIQUE (or PRIMARY) KEY

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* skip analysis of generated column data type in unique key

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
2021-05-24 20:16:49 +02:00
Tim Vaillancourt
c71dbf9ef3
Copy auto increment (#967)
* v1.1.0

* WIP: copying AUTO_INCREMENT value to ghost table
Initial commit: towards setting up a test suite

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* greping for 'expect_table_structure' content

* Adding simple test for 'expect_table_structure' scenario

* adding tests for AUTO_INCREMENT value after row deletes. Should initially fail

* clear event beforehand

* parsing AUTO_INCREMENT from alter query, reading AUTO_INCREMENT from original table, applying AUTO_INCREMENT value onto ghost table if applicable and user has not specified AUTO_INCREMENT in alter statement

* support GetUint64

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* minor update to test

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* adding test for user defined AUTO_INCREMENT statement

Co-authored-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
2021-05-14 15:32:56 +02:00
Tim Vaillancourt
0dc64757eb
Merge branch 'master' into cathal/safer_cut_over 2021-05-07 15:13:29 +02:00
Fan()
74f807103e
Update inspect.go 2021-04-29 14:33:10 +08:00
Fan()
f40f14b9ee
Update inspect.go
fix https://github.com/github/gh-ost/issues/961
2021-04-29 13:43:24 +08:00
Tim Vaillancourt
157dba920c Add mysql port and version 2021-04-03 23:24:29 +02:00
Tim Vaillancourt
23a421cef7 Add 'Hostname:' prefix to output 2021-04-03 22:34:20 +02:00
Tim Vaillancourt
5e9b913035
Merge branch 'master' into add-server-cmd-hosts 2021-04-02 16:59:56 +02:00
Tim Vaillancourt
123b46f9bb Split into 'applier' and 'inspector' commands 2021-04-02 16:57:13 +02:00
Tim Vaillancourt
cffb523bad Fix help typo 2021-04-02 02:02:08 +02:00
Tim Vaillancourt
7ea47cbfb5 Fix order 2021-04-02 01:58:59 +02:00
Tim Vaillancourt
094d11d722 Use a single line 2021-04-02 01:58:06 +02:00
Tim Vaillancourt
2fb524f43a Adds 'hosts' command to server 2021-04-02 01:50:11 +02:00
Rashiq
d3bf3cde4d
Merge branch 'master' into cathal/safer_cut_over 2021-03-08 20:59:14 +01:00
Rashiq
db513872c1
Merge branch 'master' into iss909 2021-03-08 18:20:03 +01:00
Tim Vaillancourt
048d5838db
Fix whitespace after merge-conflict fix 2021-02-08 13:37:39 +01:00
Tim Vaillancourt
72ccd0b0d0
Fix whitespace after merge-conflict fix 2021-02-08 13:36:59 +01:00
Tim Vaillancourt
edbc8d6733
Merge branch 'master' into improve-conn-log 2021-02-08 13:35:30 +01:00
Cathal Coffey
3135a25c1f HeartbeatLag must be < than --max-lag-millis and --cut-over-lock-timeout-seconds to continue 2021-02-07 14:31:40 +00:00
Cathal Coffey
503b7b0d6c Consolidate the two sleepWhileTrue loops 2021-02-07 13:52:59 +00:00
Cathal Coffey
d5c2414893 Move 'heartbeat is too high' back to Debug logs again 2021-02-05 10:12:54 +00:00
Cathal Coffey
48ce0873de Store lastHeartbeatOnChangelogTime instead of CurrentHeartbeatLag 2021-02-03 10:03:45 +00:00
Cathal Coffey
4efd156759 Move 'heartbeat is too high' to Debug logs 2021-02-02 09:17:31 +00:00
Cathal Coffey
8a26c9ebf4 Don't cut-over until it is safe to do so 2021-02-01 10:14:05 +00:00
Cathal Coffey
a4218cd6f4 Progress should print HeartbeatLag 2021-01-31 18:27:34 +00:00
Cathal Coffey
8aee288fd7 Handle onChangelogHeartbeatEvent and update CurrentHeartbeatLag 2021-01-31 18:24:26 +00:00
Cathal Coffey
7207bc146a Make it easier to handle different onChangelogEvents 2021-01-31 18:23:09 +00:00
thsun
d0f0b95cf8 rm main 2021-01-13 17:21:21 +08:00
Tim Vaillancourt
bf408b0d0e
Merge branch 'master' into master 2020-12-16 12:53:59 +01:00
Tim Vaillancourt
88e59dd2a3
Merge branch 'master' into master 2020-10-22 23:22:32 +02:00
Tim Vaillancourt
8c4cd10d79
Merge branch 'master' into ipv6 2020-10-22 11:39:46 +02:00
Tim Vaillancourt
0e2d33ad86 Merge in https://github.com/github/gh-ost/pull/755 2020-10-20 16:08:49 +02:00
Tim Vaillancourt
f268259f5e
Merge branch 'master' into handle_driver_timeout_error 2020-08-19 22:04:55 +02:00
Tim Vaillancourt
4bbc8deb77
Fix tabs from merge conflict 2020-08-13 15:50:38 +02:00
Tim Vaillancourt
5b6da5b7f3
Merge branch 'master' into handle_driver_timeout_error 2020-08-13 15:49:34 +02:00
Shlomi Noach
c9249f2b71 Updating and using AlterTableOptions 2020-07-23 11:38:05 +03:00
Shlomi Noach
f482356a94
Merge branch 'master' into parse-alter-statement 2020-07-22 12:36:59 +03:00