Commit Graph

606 Commits

Author SHA1 Message Date
Shlomi Noach
bb06e817fe compute sharedVirtualColumns and take it into account while sanity-testing chosen uniqueKeyColumns
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
2021-07-08 15:30:11 +03:00
Shlomi Noach
4610fa771f ' GENERATED' was correct, after all
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
2021-07-08 13:48:35 +03:00
Shlomi Noach
faededefcd GENERATED colunm as part of PK: handling UPDATE scenario
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
2021-07-06 08:56:06 +03:00
Shlomi Noach
54ba3cb5b3 Merge branch 'master' into switch-to-rbr-only-on-replica 2021-06-27 10:46:10 +03:00
Tim Vaillancourt
47d49c6b92
Add go mod (#935)
* Add a go.mod file

* run go mod vendor again

* Move to a well-supported ini file reader

* Remove GO111MODULE=off

* Use go 1.16

* Rename github.com/outbrain/golib -> github.com/openark/golib

* Remove *.go-e files

* Fix for `strconv.ParseInt: parsing "": invalid syntax` error

* Add test for '[osc]' section

Co-authored-by: Nate Wernimont <nate.wernimont@workiva.com>
2021-06-24 20:19:37 +02:00
Shlomi Noach
47f14a4d40 Only allow -switch-to-rbr when inspecting replica
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
2021-06-20 08:55:07 +03:00
Dirkjan Bussink
40acde0222
Set the ServerName for TLS configuration (#988)
When TLS hostname validation used for the MySQL connection, the
ServerName property needs to be set so that it knows which name to
validate on the certificate.

Without this option and with InsecureSkipVerify set to false, validation
will error here with a fatal error otherwise:

```
FATAL tls: either ServerName or InsecureSkipVerify must be specified in the tls.Config
```
2021-06-17 14:51:43 +02:00
Shlomi Noach
9bc508f068
Enum to varchar (#963)
* v1.1.0

* WIP: copying AUTO_INCREMENT value to ghost table
Initial commit: towards setting up a test suite

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* greping for 'expect_table_structure' content

* Adding simple test for 'expect_table_structure' scenario

* adding tests for AUTO_INCREMENT value after row deletes. Should initially fail

* clear event beforehand

* parsing AUTO_INCREMENT from alter query, reading AUTO_INCREMENT from original table, applying AUTO_INCREMENT value onto ghost table if applicable and user has not specified AUTO_INCREMENT in alter statement

* support GetUint64

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* minor update to test

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* adding test for user defined AUTO_INCREMENT statement

* Generated column as part of UNIQUE (or PRIMARY) KEY

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* skip analysis of generated column data type in unique key

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* All MySQL DBs limited to max 3 concurrent/idle connections

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* hooks: reporting GH_OST_ETA_SECONDS. ETA stored as part of migration context

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* GH_OST_ETA_NANOSECONDS

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* N/A denoted by negative value

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* ETAUnknown constant

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* Convering enum to varchar

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* test: not null

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* first attempt at setting enum-to-string right

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* fix insert query

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* store enum values, use when populating

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* apply EnumValues to mapped column

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* fix compilation error

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* gofmt

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
2021-06-10 17:17:49 +02:00
Shlomi Noach
f19f101610
hooks: reporting GH_OST_ETA_SECONDS. ETA as part of migration context (#936)
* v1.1.0

* WIP: copying AUTO_INCREMENT value to ghost table
Initial commit: towards setting up a test suite

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* greping for 'expect_table_structure' content

* Adding simple test for 'expect_table_structure' scenario

* adding tests for AUTO_INCREMENT value after row deletes. Should initially fail

* clear event beforehand

* parsing AUTO_INCREMENT from alter query, reading AUTO_INCREMENT from original table, applying AUTO_INCREMENT value onto ghost table if applicable and user has not specified AUTO_INCREMENT in alter statement

* support GetUint64

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* minor update to test

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* adding test for user defined AUTO_INCREMENT statement

* Generated column as part of UNIQUE (or PRIMARY) KEY

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* skip analysis of generated column data type in unique key

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* All MySQL DBs limited to max 3 concurrent/idle connections

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* hooks: reporting GH_OST_ETA_SECONDS. ETA stored as part of migration context

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* GH_OST_ETA_NANOSECONDS

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* N/A denoted by negative value

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* ETAUnknown constant

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
2021-05-31 14:15:51 +02:00
Shlomi Noach
5e19d87b1e merge from upstream, resolve conflicts
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
2021-05-31 12:09:11 +03:00
Shlomi Noach
c41823ecc9
All MySQL DBs limited to max 3 concurrent/idle connections #15 (#931)
* v1.1.0

* WIP: copying AUTO_INCREMENT value to ghost table
Initial commit: towards setting up a test suite

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* greping for 'expect_table_structure' content

* Adding simple test for 'expect_table_structure' scenario

* adding tests for AUTO_INCREMENT value after row deletes. Should initially fail

* clear event beforehand

* parsing AUTO_INCREMENT from alter query, reading AUTO_INCREMENT from original table, applying AUTO_INCREMENT value onto ghost table if applicable and user has not specified AUTO_INCREMENT in alter statement

* support GetUint64

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* minor update to test

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* adding test for user defined AUTO_INCREMENT statement

* Generated column as part of UNIQUE (or PRIMARY) KEY

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* skip analysis of generated column data type in unique key

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* All MySQL DBs limited to max 3 concurrent/idle connections

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
2021-05-27 20:00:58 +02:00
Shlomi Noach
36c669dd75
Generated column as part of UNIQUE (or PRIMARY) KEY (#919)
* v1.1.0

* WIP: copying AUTO_INCREMENT value to ghost table
Initial commit: towards setting up a test suite

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* greping for 'expect_table_structure' content

* Adding simple test for 'expect_table_structure' scenario

* adding tests for AUTO_INCREMENT value after row deletes. Should initially fail

* clear event beforehand

* parsing AUTO_INCREMENT from alter query, reading AUTO_INCREMENT from original table, applying AUTO_INCREMENT value onto ghost table if applicable and user has not specified AUTO_INCREMENT in alter statement

* support GetUint64

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* minor update to test

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* adding test for user defined AUTO_INCREMENT statement

* Generated column as part of UNIQUE (or PRIMARY) KEY

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* skip analysis of generated column data type in unique key

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
2021-05-24 20:16:49 +02:00
Tim Vaillancourt
c71dbf9ef3
Copy auto increment (#967)
* v1.1.0

* WIP: copying AUTO_INCREMENT value to ghost table
Initial commit: towards setting up a test suite

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* greping for 'expect_table_structure' content

* Adding simple test for 'expect_table_structure' scenario

* adding tests for AUTO_INCREMENT value after row deletes. Should initially fail

* clear event beforehand

* parsing AUTO_INCREMENT from alter query, reading AUTO_INCREMENT from original table, applying AUTO_INCREMENT value onto ghost table if applicable and user has not specified AUTO_INCREMENT in alter statement

* support GetUint64

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* minor update to test

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* adding test for user defined AUTO_INCREMENT statement

Co-authored-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
2021-05-14 15:32:56 +02:00
Tim Vaillancourt
0dc64757eb
Merge branch 'master' into cathal/safer_cut_over 2021-05-07 15:13:29 +02:00
Shlomi Noach
82bdf066e9 fix compilation error
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
2021-05-02 20:59:50 +03:00
Shlomi Noach
6e5b665c55 apply EnumValues to mapped column
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
2021-05-02 20:56:02 +03:00
Shlomi Noach
e80ddb42c9 store enum values, use when populating
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
2021-05-02 20:44:39 +03:00
Shlomi Noach
95ee9e2144 fix insert query
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
2021-05-02 19:37:25 +03:00
Shlomi Noach
939b898ea9 first attempt at setting enum-to-string right
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
2021-05-02 19:28:19 +03:00
Fan()
74f807103e
Update inspect.go 2021-04-29 14:33:10 +08:00
Fan()
f40f14b9ee
Update inspect.go
fix https://github.com/github/gh-ost/issues/961
2021-04-29 13:43:24 +08:00
Tim Vaillancourt
157dba920c Add mysql port and version 2021-04-03 23:24:29 +02:00
Tim Vaillancourt
23a421cef7 Add 'Hostname:' prefix to output 2021-04-03 22:34:20 +02:00
Tim Vaillancourt
5e9b913035
Merge branch 'master' into add-server-cmd-hosts 2021-04-02 16:59:56 +02:00
Tim Vaillancourt
123b46f9bb Split into 'applier' and 'inspector' commands 2021-04-02 16:57:13 +02:00
Tim Vaillancourt
cffb523bad Fix help typo 2021-04-02 02:02:08 +02:00
Tim Vaillancourt
7ea47cbfb5 Fix order 2021-04-02 01:58:59 +02:00
Tim Vaillancourt
094d11d722 Use a single line 2021-04-02 01:58:06 +02:00
Tim Vaillancourt
2fb524f43a Adds 'hosts' command to server 2021-04-02 01:50:11 +02:00
Rashiq
d3bf3cde4d
Merge branch 'master' into cathal/safer_cut_over 2021-03-08 20:59:14 +01:00
Rashiq
db513872c1
Merge branch 'master' into iss909 2021-03-08 18:20:03 +01:00
Shlomi Noach
b688c58a45 ETAUnknown constant
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
2021-03-07 14:16:04 +02:00
Shlomi Noach
76b9c16a68 N/A denoted by negative value
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
2021-03-07 11:27:50 +02:00
Shlomi Noach
51719a2b76 GH_OST_ETA_NANOSECONDS
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
2021-03-07 11:11:50 +02:00
Shlomi Noach
54000ab516 hooks: reporting GH_OST_ETA_SECONDS. ETA stored as part of migration context
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
2021-03-07 10:47:40 +02:00
thsun
40bc5aedf2 enhance processing efficiency 2021-02-26 14:23:24 +08:00
thsun
960d801276 canonical code 2021-02-24 17:18:21 +08:00
Shlomi Noach
dea8d54be0
Merge branch 'master' into limit-mysql-connetions 2021-02-22 12:31:07 +02:00
Shlomi Noach
710c9ddda5 All MySQL DBs limited to max 3 concurrent/idle connections
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
2021-02-18 10:44:47 +02:00
Tim Vaillancourt
048d5838db
Fix whitespace after merge-conflict fix 2021-02-08 13:37:39 +01:00
Tim Vaillancourt
72ccd0b0d0
Fix whitespace after merge-conflict fix 2021-02-08 13:36:59 +01:00
Tim Vaillancourt
edbc8d6733
Merge branch 'master' into improve-conn-log 2021-02-08 13:35:30 +01:00
Shlomi Noach
4a36e246c0
Merge pull request #14 from ccoffey/cathal/safer_cut_over
Cut-over should wait for heartbeat lag to be low enough to succeed
2021-02-07 17:41:59 +02:00
Cathal Coffey
3135a25c1f HeartbeatLag must be < than --max-lag-millis and --cut-over-lock-timeout-seconds to continue 2021-02-07 14:31:40 +00:00
Cathal Coffey
503b7b0d6c Consolidate the two sleepWhileTrue loops 2021-02-07 13:52:59 +00:00
Cathal Coffey
d5c2414893 Move 'heartbeat is too high' back to Debug logs again 2021-02-05 10:12:54 +00:00
Cathal Coffey
48ce0873de Store lastHeartbeatOnChangelogTime instead of CurrentHeartbeatLag 2021-02-03 10:03:45 +00:00
Cathal Coffey
4efd156759 Move 'heartbeat is too high' to Debug logs 2021-02-02 09:17:31 +00:00
Cathal Coffey
8a26c9ebf4 Don't cut-over until it is safe to do so 2021-02-01 10:14:05 +00:00
Cathal Coffey
a4218cd6f4 Progress should print HeartbeatLag 2021-01-31 18:27:34 +00:00