Commit Graph

95 Commits

Author SHA1 Message Date
Abeyu M
f4676bf463 implement a logging interface 2019-10-07 11:10:36 -04:00
Shlomi Noach
93b88e262a
Auto-merged master into gh-ost/patch-1 on deployment 2019-01-23 08:13:55 +02:00
twotwotwo
afa108f8fa
inspect: remove redundant join conditions
The TABLE_SCHEMA and TABLE_NAME are already guaranteed to have the same value in COLUMNS and UNIQUES because of the WHEREs in each query. Dropping it from the ON clause makes it complete much faster.

On (at least) MySQL 5.6 installs with thousands of tables, this query completes in a few seconds without the additional join conditions, but takes more than a minute with it.
2019-01-22 12:26:55 -08:00
Shlomi Noach
d32304520c
Merge branch 'master' into minor_changes 2019-01-14 10:45:27 +02:00
Shlomi Noach
419794e16d
Merge branch 'master' into master 2018-12-25 07:20:47 +02:00
黄恒
29e3d48c36
Merge branch 'master' into master 2018-12-11 13:12:47 +08:00
Eduardo Nunes
5f15f8a539 Remove not needed call by grouping columnLists
There is no need to call `applyColumnTypes` more than once for the same
`databaseName` and `tableName`, it is just move the additional
`columnList` to the first call.
2018-11-01 14:43:57 +01:00
Eduardo Nunes
dfc8aa28f4 Ignore not available columns when applying column types
The given `columnLists` may not contain all columns available in the
given table. This patch prevents the code to fail. Before this patch the
code was panicking whenever it was trying to set a value into a `nil`
object, e.g. `columnList.GetColumn('non-existant').Type = SomeType`.

For some reason the application was not completely failing but as a
side-effect any column after the non-existant column would never get its
Type properly set.
2018-11-01 14:33:11 +01:00
Shlomi Noach
97c1e72dbe
Merge branch 'master' into minor_changes 2018-09-17 08:58:04 +03:00
Shlomi Noach
32f1cf1dbe Support for GCP (Google Cloud Platform) 2018-09-02 09:51:40 +03:00
钟越
ef686a4e7b Support database with underscore 2018-08-31 09:31:58 +08:00
MOON_CLJ
14eda7efe0 fix GetReplicationLag not used args 2018-05-23 13:13:25 +08:00
MOON_CLJ
21d455013e minor changes and typo correction 2018-05-23 13:13:25 +08:00
Shlomi Noach
db871b42c4 Support for GENERATED (aka virtual) columns 2018-05-22 12:36:52 +03:00
Shlomi Noach
160e7a11f5 Enforcing binlog_row_image=FULL 2018-04-24 05:56:03 -07:00
zj118228
d3b98d9e74 Refactor the code and remove the global variable of
support-aliyun-rds branch.
2018-03-29 12:20:41 +08:00
zj118228
6f3d54a136 Aliyun RDS hide some mysql variables for secrity so gh-ost will get
ilegal values. Add --aliyun-rds flag to avoid getting them.
2018-01-23 14:44:22 +08:00
Nikhil Mathew
ec6ceffbcb Pass in a migrationContext UUID for a migration specific connections cache 2017-12-14 15:53:29 -08:00
Nikhil Mathew
538833ea84
Merge branch 'master' into nm-refactor-migration-context 2017-11-14 10:29:49 -08:00
Nikhil Mathew
2ad65ee1d7 Refactor usages of GetDB to use mysql 2017-11-10 15:44:22 -08:00
Josh Soref
6bc04eee62 spelling: thing 2017-11-08 01:02:32 +00:00
Josh Soref
961e2409f6 spelling: proceeding 2017-11-08 00:50:56 +00:00
Josh Soref
291c6d3594 spelling: entirely 2017-11-08 00:47:36 +00:00
Nikhil Mathew
84bdfdb1ad Cache DB connection pools on the migrationContext where applicable 2017-09-22 16:06:06 -07:00
Zach Moazeni
df27c5b76f Allow gh-ost to modify the server using extra port
Both Percona and Maria allow MySQL to be configured to listen on an extra port when their thread pool is enable.

* https://www.percona.com/doc/percona-server/5.7/performance/threadpool.html
* https://mariadb.com/kb/en/the-mariadb-library/thread-pool-in-mariadb-51-53/

This is valuable because if the table has a lot of traffic (read or write load), gh-ost can end up starving the thread pool as incomming connections are immediately blocked.

By using gh-ost on the extra port, MySQL locking will still behave the same, but MySQL will keep a dedicated thread for each gh-ost connection.

When doing this, it's important to inspect the extra-max-connections variable. Both Percona and Maria default to 1, so gh-ost may easily exceed with its threads.

An example local run using this

```
$ mysql -S /tmp/mysql_sandbox20393.sock -e "select @@global.port, @@global.extra_port"
+---------------+---------------------+
| @@global.port | @@global.extra_port |
+---------------+---------------------+
|         20393 |               30393 |
+---------------+---------------------+

./bin/gh-ost \
--initially-drop-ghost-table \
--initially-drop-old-table \
--assume-rbr \
--port="20395" \
--assume-master-host="127.0.0.1:30393" \
--max-load=Threads_running=25 \
--critical-load=Threads_running=1000 \
--chunk-size=1000 \
--max-lag-millis=1500 \
--user="gh-ost" \
--password="gh-ost" \
--database="test" \
--table="mytable" \
--verbose \
--alter="ADD mynewcol decimal(11,2) DEFAULT 0.0 NOT NULL" \
--exact-rowcount \
--concurrent-rowcount \
--default-retries=120 \
--panic-flag-file=/tmp/ghost.panic.flag \
--postpone-cut-over-flag-file=/tmp/ghost.postpone.flag \
--execute
```
2017-09-20 16:05:20 -04:00
Shlomi Noach
a2847015d6 Merge branch 'master' into nm-refactor-migration-context 2017-09-10 08:08:39 +03:00
Shlomi Noach
78c1e0432f Merge branch 'master' into checking-thresholds 2017-09-06 08:07:35 +03:00
Shlomi Noach
e2171e0162 Validating password length 2017-09-03 10:27:04 +03:00
Shlomi Noach
3437cf44c4 Validating shared key column types 2017-09-03 09:57:26 +03:00
Nikhil Mathew
0a7e713e9f Ensure cleanup happens, even on error 2017-08-28 15:53:47 -07:00
Shlomi Noach
2174588ebf Supporting JSON types 2017-08-22 11:51:20 +03:00
Nikhil Mathew
982b8eede9 Refactor global migration context 2017-08-08 13:36:54 -07:00
Shlomi Noach
effae08fbf supporting change of column name-case 2017-07-12 12:33:32 +03:00
Shlomi Noach
8a0f1413eb dropped columns are not 'shared' and no data copy attempted for such columns 2017-04-23 08:38:35 +03:00
Shlomi Noach
f11f6f978f mitigating cut-over/write race condition 2017-02-24 14:48:10 -07:00
Shlomi Noach
ed5dd7f049 Collecting and presenting MySQL versions of applier and inspector 2017-02-07 09:41:33 +02:00
Shlomi Noach
a07a6f8cf5 fixing mediumint unsigned problem 2017-01-10 09:57:42 +02:00
Shlomi Noach
fc831b0548 Reading replication lag via _changelog_ table, also on control replicas 2016-12-26 21:31:35 +02:00
Shlomi Noach
d12c0265b0 sleeping 500ms after 'START SLAVE' 2016-12-13 12:27:20 +01:00
Shlomi Noach
1d84cb933c fix: bailing out on no PRIMARY/UNIQUE KEY 2016-12-11 02:19:19 +01:00
Shlomi Noach
35eeb56032 improved log/error messages 2016-12-05 13:41:49 +01:00
rj03hou
8f02ab0fed check the slave status when find recursive find the master, so support if the dba using reset slave instead of reset slave all. 2016-12-05 19:42:16 +08:00
Shlomi Noach
7126b28169 support for --skip-foreign-key-checks 2016-11-21 09:18:40 +01:00
Shlomi Noach
c2d4f624af simplified code 2016-11-03 12:14:53 +01:00
Shlomi Noach
88ffb75b8c reading and reporting replication lag before waiting on initial replication event 2016-11-02 12:48:35 +01:00
Shlomi Noach
bb22431b83 fixed log_slave_updates check logic 2016-10-31 10:25:34 +01:00
Shlomi Noach
ac6159791d merged master, resolved conflicts 2016-10-26 09:57:59 +02:00
Shlomi Noach
b696106cb6 fixing bug introduced for charset and timezone tests 2016-10-20 15:05:47 +02:00
Shlomi Noach
bf92eec214 validating table structure on applier and migrator
- reading column list on applier
- comparing original table on applier and migrator, expecting exact column list
- or else bailing out
2016-10-20 11:29:30 +02:00
Shlomi Noach
82a76ae879 Merge branch 'master' into enum-sort-test 2016-10-20 10:16:02 +02:00