octoleo/gh-ost
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
1,546 Commits 51 Branches 51 Tags 24 MiB
8eb9708755
Commit Graph

446 Commits

Author SHA1 Message Date
Shlomi Noach
60398e774f
Merge branch 'master' into add-dry-run-env 2018-10-02 14:08:08 +03:00
Shlomi Noach
6284a34823
Merge branch 'master' into ipv6 2018-10-02 08:12:15 +03:00
Shlomi Noach
7bd338faf7 Fix to retry logic in row copy 2018-09-20 10:49:21 +03:00
Jacob Bednarz
09a005d79e
Populate the env with dry run status 
Updates the `applyEnvironmentVariables` function to populate whether or
not the current execution context is running as a dry run or not which
can then be used in hooks.
 2018-09-18 14:41:26 +10:00
Shlomi Noach
04c0be6137
Merge branch 'master' into ipv6 2018-09-17 09:35:18 +03:00
Shlomi Noach
97c1e72dbe
Merge branch 'master' into minor_changes 2018-09-17 08:58:04 +03:00
Shlomi Noach
695e32d41b
Merge branch 'master' into dev_copyrow_hangs 2018-09-17 08:42:03 +03:00
Shlomi Noach
49b80df27b Parsing ipv6 addresses 2018-09-16 11:44:52 +03:00
Shlomi Noach
32f1cf1dbe Support for GCP (Google Cloud Platform) 2018-09-02 09:51:40 +03:00
钟越
ef686a4e7b Support database with underscore 2018-08-31 09:31:58 +08:00
wangfei
c7dff99a19 fix copyRowsFunc hangs bug 2018-07-26 03:02:12 +08:00
MOON_CLJ
14eda7efe0 fix GetReplicationLag not used args 2018-05-23 13:13:25 +08:00
MOON_CLJ
21d455013e minor changes and typo correction 2018-05-23 13:13:25 +08:00
Shlomi Noach
db871b42c4 Support for GENERATED (aka virtual) columns 2018-05-22 12:36:52 +03:00
Shlomi Noach
dfeefed5dc avoid prepared statements for changelog table 2018-05-20 13:46:33 +03:00
Shlomi Noach
fb00a13871 Rejecting RENAME TO|AS 2018-05-06 11:19:03 +03:00
Shlomi Noach
160e7a11f5 Enforcing binlog_row_image=FULL 2018-04-24 05:56:03 -07:00
Shlomi Noach
3dc4930ae3
Merge branch 'master' into support-aliyun-rds 2018-04-16 14:17:03 +03:00
zj118228
d3b98d9e74 Refactor the code and remove the global variable of 
support-aliyun-rds branch.
 2018-03-29 12:20:41 +08:00
Kurt Kotzur
15e7417fc5
update comment 2018-03-19 12:29:49 -07:00
Kurt Kotzur
117b197b2a
Obey defaultNumRetries in retryOperationWithExponentialBackoff; name max interval flag more generically (#2) 
* rename flags, obey defaultNumRetries

* capitalization fixes

* fix flag description typo

* fix sleep algorithm
 2018-03-19 12:26:46 -07:00
Kurt Kotzur
628983f78f
🎨 2018-03-16 12:42:38 -07:00
Kurt Kotzur
64f66c4abb
add back nonFatalHint, fix type mismatches 2018-03-16 10:54:37 -07:00
Kurt Kotzur
bd62b19b0b
remove nonFatalHint argument from retryOperation; add retryOperationWithExponentialBackoff and conditionally use it 2018-03-15 17:41:15 -07:00
Shlomi Noach
bf01b31795
Merge branch 'master' into support-aliyun-rds 2018-03-07 16:39:55 +02:00
Shlomi Noach
9ae9cef17a support quoted values in interactive commands 2018-02-25 19:17:37 +02:00
zj118228
6f3d54a136 Aliyun RDS hide some mysql variables for secrity so gh-ost will get 
ilegal values. Add --aliyun-rds flag to avoid getting them.
 2018-01-23 14:44:22 +08:00
Shlomi Noach
02579075d0 Call the throttler.Teardown 2018-01-11 08:03:09 +02:00
Nikhil Mathew
ec6ceffbcb Pass in a migrationContext UUID for a migration specific connections cache 2017-12-14 15:53:29 -08:00
Nikhil Mathew
fac1ba7026 Throttler teardown 2017-12-14 15:14:13 -08:00
Shlomi Noach
0530b09700
Merge branch 'master' into nm-refactor-migration-context 2017-12-07 13:59:12 +02:00
Shlomi Noach
58c381f0a0 some cleanup 2017-11-21 09:14:04 +02:00
Shlomi Noach
203ea6c2cd refactor, simplifid code 2017-11-20 10:24:28 +02:00
Shlomi Noach
3898d49f6c supporting update to columns of migration key 2017-11-20 08:17:20 +02:00
Nikhil Mathew
538833ea84
Merge branch 'master' into nm-refactor-migration-context 2017-11-14 10:29:49 -08:00
Nikhil Mathew
2ad65ee1d7 Refactor usages of GetDB to use mysql 2017-11-10 15:44:22 -08:00
Shlomi Noach
8a59d7e823 failing on PK value change 2017-11-08 11:11:17 +02:00
Josh Soref
6bc04eee62 spelling: thing 2017-11-08 01:02:32 +00:00
Josh Soref
3196c890f5 spelling: running 2017-11-08 01:01:39 +00:00
Josh Soref
4c93b7ac6d spelling: response 2017-11-08 00:55:09 +00:00
Josh Soref
961e2409f6 spelling: proceeding 2017-11-08 00:50:56 +00:00
Josh Soref
6a8d97c13f spelling: progress 2017-11-08 00:50:48 +00:00
Josh Soref
867e6a28b3 spelling: merely 2017-11-08 00:49:51 +00:00
Josh Soref
e2df0b80eb spelling: iterating 2017-11-08 00:49:23 +00:00
Josh Soref
2b430e3cb6 spelling: ignoring 2017-11-08 00:49:06 +00:00
Josh Soref
e608c16567 spelling: explicitly 2017-11-08 00:47:54 +00:00
Josh Soref
291c6d3594 spelling: entirely 2017-11-08 00:47:36 +00:00
Josh Soref
418462a6ca spelling: dropping 2017-11-08 00:47:15 +00:00
Josh Soref
8901e6eec2 spelling: decision 2017-11-08 00:46:04 +00:00
Josh Soref
d29993ffda spelling: changelog 2017-11-08 00:45:19 +00:00
Josh Soref
4dda9cf930 spelling: aggressive 2017-11-08 00:44:30 +00:00
Akshay Chhajed
ccb7654235 Improved connection type logging 2017-10-29 19:53:32 +05:30
Shlomi Noach
087a6cc0cb Merge branch 'master' into touch-postpone-flag-file 2017-10-17 16:02:11 +03:00
Nikhil Mathew
f06e23282e Merge branch 'master' into nm-refactor-migration-context 2017-10-02 10:31:57 -07:00
Shlomi Noach
aff7413624 clearer error message when not able to create postpone-cut-over-flag-file 2017-10-02 16:00:58 +03:00
Shlomi Noach
c2186db527 Merge branch 'master' into touch-postpone-flag-file 2017-10-02 15:55:21 +03:00
Shlomi Noach
9890e66755 Merge pull request #491 from zmoazeni/enable-extra-port 
Allow gh-ost to modify the server using extra port
 2017-10-02 14:26:59 +03:00
Nikhil Mathew
84bdfdb1ad Cache DB connection pools on the migrationContext where applicable 2017-09-22 16:06:06 -07:00
Nikhil Mathew
5788ab5347 CR Revisions: Make concurrent operations safe 2017-09-22 15:17:58 -07:00
dcadevil
351bfe1a24 Correct spelling for initiateStreaming() comment. 2017-09-21 11:36:27 +08:00
Zach Moazeni
df27c5b76f Allow gh-ost to modify the server using extra port 
Both Percona and Maria allow MySQL to be configured to listen on an extra port when their thread pool is enable.

* https://www.percona.com/doc/percona-server/5.7/performance/threadpool.html
* https://mariadb.com/kb/en/the-mariadb-library/thread-pool-in-mariadb-51-53/

This is valuable because if the table has a lot of traffic (read or write load), gh-ost can end up starving the thread pool as incomming connections are immediately blocked.

By using gh-ost on the extra port, MySQL locking will still behave the same, but MySQL will keep a dedicated thread for each gh-ost connection.

When doing this, it's important to inspect the extra-max-connections variable. Both Percona and Maria default to 1, so gh-ost may easily exceed with its threads.

An example local run using this

```
$ mysql -S /tmp/mysql_sandbox20393.sock -e "select @@global.port, @@global.extra_port"
+---------------+---------------------+
| @@global.port | @@global.extra_port |
+---------------+---------------------+
|         20393 |               30393 |
+---------------+---------------------+

./bin/gh-ost \
--initially-drop-ghost-table \
--initially-drop-old-table \
--assume-rbr \
--port="20395" \
--assume-master-host="127.0.0.1:30393" \
--max-load=Threads_running=25 \
--critical-load=Threads_running=1000 \
--chunk-size=1000 \
--max-lag-millis=1500 \
--user="gh-ost" \
--password="gh-ost" \
--database="test" \
--table="mytable" \
--verbose \
--alter="ADD mynewcol decimal(11,2) DEFAULT 0.0 NOT NULL" \
--exact-rowcount \
--concurrent-rowcount \
--default-retries=120 \
--panic-flag-file=/tmp/ghost.panic.flag \
--postpone-cut-over-flag-file=/tmp/ghost.postpone.flag \
--execute
```
 2017-09-20 16:05:20 -04:00
Shlomi Noach
a2847015d6 Merge branch 'master' into nm-refactor-migration-context 2017-09-10 08:08:39 +03:00
Shlomi Noach
78c1e0432f Merge branch 'master' into checking-thresholds 2017-09-06 08:07:35 +03:00
Shlomi Noach
e2171e0162 Validating password length 2017-09-03 10:27:04 +03:00
Shlomi Noach
3437cf44c4 Validating shared key column types 2017-09-03 09:57:26 +03:00
Christian Joudrey
2646a0a997
Remove unused acceptSignals in migrator 2017-09-02 21:59:09 -04:00
Nikhil Mathew
0a7e713e9f Ensure cleanup happens, even on error 2017-08-28 15:53:47 -07:00
Nikhil Mathew
e4bb70df43 Use change from outbrain/golib PR 2017-08-28 15:19:05 -07:00
Nikhil Mathew
3b21f4db37 Close remaining DB connections 2017-08-28 14:05:15 -07:00
Shlomi Noach
c9f2da82f0 Merge branch 'master' into json-57-test 2017-08-27 08:01:02 +03:00
Shlomi Noach
2174588ebf Supporting JSON types 2017-08-22 11:51:20 +03:00
Shlomi Noach
79e4f1cdbe detect range end based on OFFSET 2017-08-21 08:12:41 +03:00
Nikhil Mathew
a1473dfb58 Stop infinite goroutines 2017-08-08 15:31:25 -07:00
Nikhil Mathew
7412f42371 Remove no longer needed acceptSignals func 2017-08-08 13:44:53 -07:00
Nikhil Mathew
b399533d4f Simplify GoMySQLReader 2017-08-08 13:41:46 -07:00
Nikhil Mathew
982b8eede9 Refactor global migration context 2017-08-08 13:36:54 -07:00
Shlomi Noach
de0ebcbbb0 documentation 2017-07-19 16:48:22 +03:00
Shlomi Noach
6da0b8af3d Dynamic DML batch size; apadting buffer size to match 2017-07-19 16:44:18 +03:00
Shlomi Noach
4a3a5a1ce1 Merge branch 'master' into case-insensitive-columns 2017-07-16 06:51:39 +03:00
Shlomi Noach
effae08fbf supporting change of column name-case 2017-07-12 12:33:32 +03:00
Shlomi Noach
e6dd2c90d2 responding to errors in rowcopy 2017-07-11 13:39:11 +03:00
Shlomi Noach
a8932ee1b8 freno-aware http-throttler 2017-07-04 10:28:09 +03:00
Shlomi Noach
36bcc031ce Merge branch 'master' into fix-variables-spelling 2017-06-29 20:53:49 +03:00
Shlomi Noach
cb3c7f3e20 Merge branch 'master' into touch-postpone-flag-file 2017-06-11 08:57:45 +03:00
Shlomi Noach
8da0f60582 fixed critical-load check for hibernation 2017-05-24 10:53:00 +03:00
Shlomi Noach
ad47f7c147 throttling just prior to leaving hibernation, so as to allow re-throttle checks to apply 2017-05-24 10:42:47 +03:00
Shlomi Noach
3955a6d67f hibernate on critical-load 2017-05-24 08:32:13 +03:00
Shlomi Noach
aca288e88d -postpone-cut-over-flag-file implies touching indicated file 2017-05-07 14:58:18 +03:00
Shlomi Noach
eaa3489fe0 Merge branch 'master' into server-report-coordinates 2017-05-01 07:50:13 +03:00
Shlomi Noach
79cee99f57 support for 'coordinates' command 2017-04-28 15:50:51 -07:00
Jacob Bednarz
0243e76b33
s/applyEnvironmentVairables/applyEnvironmentVariables 2017-04-24 07:45:21 +10:00
Shlomi Noach
e2a6b857d0 Merge branch 'master' into drop-add-same-column 2017-04-23 20:26:31 +03:00
Shlomi Noach
8a0f1413eb dropped columns are not 'shared' and no data copy attempted for such columns 2017-04-23 08:38:35 +03:00
Shlomi Noach
3092e9c5c0 error and nil checks on socket connection 2017-04-13 08:27:42 +03:00
Shlomi Noach
c413d508cc HEAD instead of GET 2017-03-26 13:12:56 +03:00
Shlomi Noach
23ce390d69 supporting throttle-http 2017-03-26 13:10:34 +03:00
Shlomi Noach
7517238bd3 Merge branch 'master' into fix-infinite-cutover-loop 2017-03-06 21:50:26 +02:00
Shlomi Noach
91521f9357 Merge branch 'master' into old-table-name-timestamp 2017-03-05 18:10:14 +02:00
Shlomi Noach
f11f6f978f mitigating cut-over/write race condition 2017-02-24 14:48:10 -07:00
Shlomi Noach
06c909bd10 Validating table name length 2017-02-21 17:34:49 -07:00
Shlomi Noach
289ce46a2b Merge branch 'master' into fix-infinite-cutover-loop 2017-02-16 14:16:32 +02:00
Shlomi Noach
b289041ccb Merge branch 'master' into fix-reappearing-throttled-reasons 2017-02-08 13:11:34 +02:00
Shlomi Noach
6ecf7cc0ee Auto-merged master into interactive-command-question on deployment 2017-02-08 13:03:45 +02:00
Shlomi Noach
10edf3c063 Migration only starting after first replication lag metric collected 2017-02-07 12:13:19 +02:00
Shlomi Noach
91a3fd4a84 Merge branch 'master' into which-mysql-versions 2017-02-07 11:12:48 +02:00
Shlomi Noach
ed5dd7f049 Collecting and presenting MySQL versions of applier and inspector 2017-02-07 09:41:33 +02:00
Shlomi Noach
6fa32f6d54 cut-over failure on test-on-replica starts replication again 2017-02-07 09:31:52 +02:00
Shlomi Noach
9d47992d7e testing/running on replica: gets lags via SHOW SLAVE STATUS 2017-02-02 11:18:07 +02:00
Shlomi Noach
baee4f69f9 fixing phantom throttle-control-replicas lag result 2017-01-29 10:18:39 +02:00
Shlomi Noach
be1ab175c7 status presents with '# throttle-control-replicas count:' 2017-01-29 09:56:25 +02:00
Shlomi Noach
779e9fdd83 question (?) as argument in interactive commands 2017-01-29 09:25:29 +02:00
Shlomi Noach
a07a6f8cf5 fixing mediumint unsigned problem 2017-01-10 09:57:42 +02:00
Shlomi Noach
f7d2beb4d2 handling a non-DML event at the end of a dml-event sequence 2017-01-05 08:13:51 +02:00
Shlomi Noach
baaa255182 bailing out from onApplyEventStruct() 2017-01-04 12:42:21 +02:00
Shlomi Noach
645af21d03 extracted onApplyEventStruct() 2017-01-04 12:39:57 +02:00
Shlomi Noach
8d0faa55e3 explicit rollback in ApplyDMLEventQueries() 2017-01-04 08:44:04 +02:00
Shlomi Noach
220bf83a5b Merge branch 'master' into batch-apply-dml-events 2017-01-04 08:37:08 +02:00
Shlomi Noach
445c67635d batching DML writes, configurable --dml-batch-size 2017-01-03 14:31:19 +02:00
Shlomi Noach
3e28f462d8 Initial support for batching multiple DMLs when writing to ghost table 2017-01-03 13:44:52 +02:00
Shlomi Noach
c66356bd05 --replication-lag-query is deprecated. 2016-12-26 21:38:37 +02:00
Shlomi Noach
fc831b0548 Reading replication lag via _changelog_ table, also on control replicas 2016-12-26 21:31:35 +02:00
Shlomi Noach
eac990bc95 Merge branch 'master' into explicit-master-user-password 2016-12-15 07:58:12 +01:00
Shlomi Noach
ba2a9d9e55 support for --master-user and --master-password 2016-12-13 16:09:34 +01:00
Shlomi Noach
d12c0265b0 sleeping 500ms after 'START SLAVE' 2016-12-13 12:27:20 +01:00
Shlomi Noach
1d84cb933c fix: bailing out on no PRIMARY/UNIQUE KEY 2016-12-11 02:19:19 +01:00
Shlomi Noach
35eeb56032 improved log/error messages 2016-12-05 13:41:49 +01:00
rj03hou
8f02ab0fed check the slave status when find recursive find the master, so support if the dba using reset slave instead of reset slave all. 2016-12-05 19:42:16 +08:00
Shlomi Noach
5904084e14 Merge branch 'master' into myisam-gtid 2016-12-02 15:00:44 +01:00
rj03hou
ffbd35e180 fix TableEngine correlates to the 3rd placeholder in the template string, not the 1st 2016-12-02 11:56:29 +08:00
Shlomi Noach
9b068ec222 Merge branch 'master' into myisam-gtid 2016-12-01 09:43:38 +01:00
rj03hou
a11bec1785 If the original table is MyISAM and the default engine is Innodb, and the gtid mode is on, there will be error when execute 'LOCK TABLES tbl WRITE, tbl_magic WRITE'. If make the magic cut-over table's engine same with the original table, there will not have this problem. 2016-12-01 16:04:04 +08:00
Shlomi Noach
5119ea4d31 added tests to verify no false positives rename-column found 2016-11-29 11:08:35 +01:00
Shlomi Noach
7126b28169 support for --skip-foreign-key-checks 2016-11-21 09:18:40 +01:00
Shlomi Noach
7ab6af8f5f never throttling inside cut-over critical section 2016-11-17 17:22:13 +01:00
Shlomi Noach
b00cae11fa retry cut-over 2016-11-17 17:10:17 +01:00
Shlomi Noach
8d987b5aaf extracted parsing of ChangelogState 2016-11-17 15:56:59 +01:00
Shlomi Noach
ef874b8551 AllEventsUpToLockProcessed uses unique signature 2016-11-17 15:50:54 +01:00
Shlomi Noach
ee447ad560 waitForEventsUpToLock timeout 
more info on AllEventsUpToLockProcessed, before and after injecting/intercepting
 2016-11-17 15:20:44 +01:00
Shlomi Noach
c2d4f624af simplified code 2016-11-03 12:14:53 +01:00
Shlomi Noach
88ffb75b8c reading and reporting replication lag before waiting on initial replication event 2016-11-02 12:48:35 +01:00
Shlomi Noach
034683f482 Merge branch 'master' into close-streamer-connection 2016-11-01 12:22:21 +01:00
Shlomi Noach
b32e1e2c16 Merge branch 'master' into throttle-no-changelog-write 2016-11-01 12:20:51 +01:00
Shlomi Noach
bb22431b83 fixed log_slave_updates check logic 2016-10-31 10:25:34 +01:00
Shlomi Noach
7fa5e405d4 avoid writing heartbeat when throttle commanded by user 
when throttling on user command there really is no need for injecting heartbeat. The user commanded, therefore gh-ost complies and trusts the reasoning for throttling. What this will allow is complete quiet time. This, in turn, will allow such features as relocating via orchestrator/pseudo-gtid at time of throttling
 2016-10-27 14:51:38 +02:00
Shlomi Noach
7b63b4a275 proper cleanup of streamer connection 2016-10-27 13:52:37 +02:00
Shlomi Noach
ac6159791d merged master, resolved conflicts 2016-10-26 09:57:59 +02:00
Shlomi Noach
b696106cb6 fixing bug introduced for charset and timezone tests 2016-10-20 15:05:47 +02:00
Shlomi Noach
bf92eec214 validating table structure on applier and migrator 
- reading column list on applier
- comparing original table on applier and migrator, expecting exact column list
- or else bailing out
 2016-10-20 11:29:30 +02:00
Shlomi Noach
82a76ae879 Merge branch 'master' into enum-sort-test 2016-10-20 10:16:02 +02:00
Akshay Chhajed
559c7895ca Modified slave update check to test tungsten flag 2016-10-19 23:31:16 +05:30