Commit Graph

106 Commits

Author SHA1 Message Date
Shlomi Noach
36c669dd75
Generated column as part of UNIQUE (or PRIMARY) KEY (#919)
* v1.1.0

* WIP: copying AUTO_INCREMENT value to ghost table
Initial commit: towards setting up a test suite

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* greping for 'expect_table_structure' content

* Adding simple test for 'expect_table_structure' scenario

* adding tests for AUTO_INCREMENT value after row deletes. Should initially fail

* clear event beforehand

* parsing AUTO_INCREMENT from alter query, reading AUTO_INCREMENT from original table, applying AUTO_INCREMENT value onto ghost table if applicable and user has not specified AUTO_INCREMENT in alter statement

* support GetUint64

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* minor update to test

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* adding test for user defined AUTO_INCREMENT statement

* Generated column as part of UNIQUE (or PRIMARY) KEY

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* skip analysis of generated column data type in unique key

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
2021-05-24 20:16:49 +02:00
Tim Vaillancourt
c71dbf9ef3
Copy auto increment (#967)
* v1.1.0

* WIP: copying AUTO_INCREMENT value to ghost table
Initial commit: towards setting up a test suite

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* greping for 'expect_table_structure' content

* Adding simple test for 'expect_table_structure' scenario

* adding tests for AUTO_INCREMENT value after row deletes. Should initially fail

* clear event beforehand

* parsing AUTO_INCREMENT from alter query, reading AUTO_INCREMENT from original table, applying AUTO_INCREMENT value onto ghost table if applicable and user has not specified AUTO_INCREMENT in alter statement

* support GetUint64

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* minor update to test

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* adding test for user defined AUTO_INCREMENT statement

Co-authored-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
2021-05-14 15:32:56 +02:00
Fan()
74f807103e
Update inspect.go 2021-04-29 14:33:10 +08:00
Fan()
f40f14b9ee
Update inspect.go
fix https://github.com/github/gh-ost/issues/961
2021-04-29 13:43:24 +08:00
Rashiq
db513872c1
Merge branch 'master' into iss909 2021-03-08 18:20:03 +01:00
Tim Vaillancourt
72ccd0b0d0
Fix whitespace after merge-conflict fix 2021-02-08 13:36:59 +01:00
Tim Vaillancourt
edbc8d6733
Merge branch 'master' into improve-conn-log 2021-02-08 13:35:30 +01:00
thsun
d0f0b95cf8 rm main 2021-01-13 17:21:21 +08:00
Tim Vaillancourt
88e59dd2a3
Merge branch 'master' into master 2020-10-22 23:22:32 +02:00
Shuode Li
ae762694fb Support Azure Database for MySQL. 2019-12-17 03:47:14 +00:00
Abeyu M
f4676bf463 implement a logging interface 2019-10-07 11:10:36 -04:00
Shlomi Noach
93b88e262a
Auto-merged master into gh-ost/patch-1 on deployment 2019-01-23 08:13:55 +02:00
twotwotwo
afa108f8fa
inspect: remove redundant join conditions
The TABLE_SCHEMA and TABLE_NAME are already guaranteed to have the same value in COLUMNS and UNIQUES because of the WHEREs in each query. Dropping it from the ON clause makes it complete much faster.

On (at least) MySQL 5.6 installs with thousands of tables, this query completes in a few seconds without the additional join conditions, but takes more than a minute with it.
2019-01-22 12:26:55 -08:00
Shlomi Noach
d32304520c
Merge branch 'master' into minor_changes 2019-01-14 10:45:27 +02:00
Shlomi Noach
419794e16d
Merge branch 'master' into master 2018-12-25 07:20:47 +02:00
黄恒
29e3d48c36
Merge branch 'master' into master 2018-12-11 13:12:47 +08:00
Eduardo Nunes
5f15f8a539 Remove not needed call by grouping columnLists
There is no need to call `applyColumnTypes` more than once for the same
`databaseName` and `tableName`, it is just move the additional
`columnList` to the first call.
2018-11-01 14:43:57 +01:00
Eduardo Nunes
dfc8aa28f4 Ignore not available columns when applying column types
The given `columnLists` may not contain all columns available in the
given table. This patch prevents the code to fail. Before this patch the
code was panicking whenever it was trying to set a value into a `nil`
object, e.g. `columnList.GetColumn('non-existant').Type = SomeType`.

For some reason the application was not completely failing but as a
side-effect any column after the non-existant column would never get its
Type properly set.
2018-11-01 14:33:11 +01:00
Shlomi Noach
97c1e72dbe
Merge branch 'master' into minor_changes 2018-09-17 08:58:04 +03:00
Shlomi Noach
32f1cf1dbe Support for GCP (Google Cloud Platform) 2018-09-02 09:51:40 +03:00
钟越
ef686a4e7b Support database with underscore 2018-08-31 09:31:58 +08:00
MOON_CLJ
14eda7efe0 fix GetReplicationLag not used args 2018-05-23 13:13:25 +08:00
MOON_CLJ
21d455013e minor changes and typo correction 2018-05-23 13:13:25 +08:00
Shlomi Noach
db871b42c4 Support for GENERATED (aka virtual) columns 2018-05-22 12:36:52 +03:00
Shlomi Noach
160e7a11f5 Enforcing binlog_row_image=FULL 2018-04-24 05:56:03 -07:00
zj118228
d3b98d9e74 Refactor the code and remove the global variable of
support-aliyun-rds branch.
2018-03-29 12:20:41 +08:00
zj118228
6f3d54a136 Aliyun RDS hide some mysql variables for secrity so gh-ost will get
ilegal values. Add --aliyun-rds flag to avoid getting them.
2018-01-23 14:44:22 +08:00
Nikhil Mathew
ec6ceffbcb Pass in a migrationContext UUID for a migration specific connections cache 2017-12-14 15:53:29 -08:00
Nikhil Mathew
538833ea84
Merge branch 'master' into nm-refactor-migration-context 2017-11-14 10:29:49 -08:00
Nikhil Mathew
2ad65ee1d7 Refactor usages of GetDB to use mysql 2017-11-10 15:44:22 -08:00
Josh Soref
6bc04eee62 spelling: thing 2017-11-08 01:02:32 +00:00
Josh Soref
961e2409f6 spelling: proceeding 2017-11-08 00:50:56 +00:00
Josh Soref
291c6d3594 spelling: entirely 2017-11-08 00:47:36 +00:00
Akshay Chhajed
ccb7654235 Improved connection type logging 2017-10-29 19:53:32 +05:30
Nikhil Mathew
84bdfdb1ad Cache DB connection pools on the migrationContext where applicable 2017-09-22 16:06:06 -07:00
Zach Moazeni
df27c5b76f Allow gh-ost to modify the server using extra port
Both Percona and Maria allow MySQL to be configured to listen on an extra port when their thread pool is enable.

* https://www.percona.com/doc/percona-server/5.7/performance/threadpool.html
* https://mariadb.com/kb/en/the-mariadb-library/thread-pool-in-mariadb-51-53/

This is valuable because if the table has a lot of traffic (read or write load), gh-ost can end up starving the thread pool as incomming connections are immediately blocked.

By using gh-ost on the extra port, MySQL locking will still behave the same, but MySQL will keep a dedicated thread for each gh-ost connection.

When doing this, it's important to inspect the extra-max-connections variable. Both Percona and Maria default to 1, so gh-ost may easily exceed with its threads.

An example local run using this

```
$ mysql -S /tmp/mysql_sandbox20393.sock -e "select @@global.port, @@global.extra_port"
+---------------+---------------------+
| @@global.port | @@global.extra_port |
+---------------+---------------------+
|         20393 |               30393 |
+---------------+---------------------+

./bin/gh-ost \
--initially-drop-ghost-table \
--initially-drop-old-table \
--assume-rbr \
--port="20395" \
--assume-master-host="127.0.0.1:30393" \
--max-load=Threads_running=25 \
--critical-load=Threads_running=1000 \
--chunk-size=1000 \
--max-lag-millis=1500 \
--user="gh-ost" \
--password="gh-ost" \
--database="test" \
--table="mytable" \
--verbose \
--alter="ADD mynewcol decimal(11,2) DEFAULT 0.0 NOT NULL" \
--exact-rowcount \
--concurrent-rowcount \
--default-retries=120 \
--panic-flag-file=/tmp/ghost.panic.flag \
--postpone-cut-over-flag-file=/tmp/ghost.postpone.flag \
--execute
```
2017-09-20 16:05:20 -04:00
Shlomi Noach
a2847015d6 Merge branch 'master' into nm-refactor-migration-context 2017-09-10 08:08:39 +03:00
Shlomi Noach
78c1e0432f Merge branch 'master' into checking-thresholds 2017-09-06 08:07:35 +03:00
Shlomi Noach
e2171e0162 Validating password length 2017-09-03 10:27:04 +03:00
Shlomi Noach
3437cf44c4 Validating shared key column types 2017-09-03 09:57:26 +03:00
Nikhil Mathew
0a7e713e9f Ensure cleanup happens, even on error 2017-08-28 15:53:47 -07:00
Shlomi Noach
2174588ebf Supporting JSON types 2017-08-22 11:51:20 +03:00
Nikhil Mathew
982b8eede9 Refactor global migration context 2017-08-08 13:36:54 -07:00
Shlomi Noach
effae08fbf supporting change of column name-case 2017-07-12 12:33:32 +03:00
Shlomi Noach
8a0f1413eb dropped columns are not 'shared' and no data copy attempted for such columns 2017-04-23 08:38:35 +03:00
Shlomi Noach
f11f6f978f mitigating cut-over/write race condition 2017-02-24 14:48:10 -07:00
Shlomi Noach
ed5dd7f049 Collecting and presenting MySQL versions of applier and inspector 2017-02-07 09:41:33 +02:00
Shlomi Noach
a07a6f8cf5 fixing mediumint unsigned problem 2017-01-10 09:57:42 +02:00
Shlomi Noach
fc831b0548 Reading replication lag via _changelog_ table, also on control replicas 2016-12-26 21:31:35 +02:00
Shlomi Noach
d12c0265b0 sleeping 500ms after 'START SLAVE' 2016-12-13 12:27:20 +01:00