octoleo/gh-ost
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
1,595 Commits 51 Branches 51 Tags 24 MiB
75a346be93
Commit Graph

649 Commits

Author SHA1 Message Date
Shlomi Noach
39e73e37d0 testing context.TimestampOldTable 2017-07-19 15:22:46 +03:00
Shlomi Noach
168676bd9d ghost/changelog/del table names support max characters limit 2017-07-19 15:14:03 +03:00
Shlomi Noach
4a3a5a1ce1 Merge branch 'master' into case-insensitive-columns 2017-07-16 06:51:39 +03:00
Shlomi Noach
effae08fbf supporting change of column name-case 2017-07-12 12:33:32 +03:00
Shlomi Noach
e6dd2c90d2 responding to errors in rowcopy 2017-07-11 13:39:11 +03:00
Shlomi Noach
81403699b7 Merge branch 'master' into throttle-http-freno-aware 2017-07-05 10:16:56 +03:00
Shlomi Noach
a8932ee1b8 freno-aware http-throttler 2017-07-04 10:28:09 +03:00
Shlomi Noach
f9788a61ed Merge branch 'master' into dml-batch-size-cap 2017-07-04 08:54:49 +03:00
Shlomi Noach
095863b774 dml-batch-size capped at 1000 2017-07-04 08:50:25 +03:00
Shlomi Noach
36bcc031ce Merge branch 'master' into fix-variables-spelling 2017-06-29 20:53:49 +03:00
Shlomi Noach
e8b384e43c supporting greater batch size 2017-06-28 19:35:55 +03:00
Shlomi Noach
cb3c7f3e20 Merge branch 'master' into touch-postpone-flag-file 2017-06-11 08:57:45 +03:00
Shlomi Noach
8da0f60582 fixed critical-load check for hibernation 2017-05-24 10:53:00 +03:00
Shlomi Noach
ad47f7c147 throttling just prior to leaving hibernation, so as to allow re-throttle checks to apply 2017-05-24 10:42:47 +03:00
Shlomi Noach
3955a6d67f hibernate on critical-load 2017-05-24 08:32:13 +03:00
Shlomi Noach
aca288e88d -postpone-cut-over-flag-file implies touching indicated file 2017-05-07 14:58:18 +03:00
Shlomi Noach
eaa3489fe0 Merge branch 'master' into server-report-coordinates 2017-05-01 07:50:13 +03:00
Shlomi Noach
79cee99f57 support for 'coordinates' command 2017-04-28 15:50:51 -07:00
Jacob Bednarz
0243e76b33
s/applyEnvironmentVairables/applyEnvironmentVariables 2017-04-24 07:45:21 +10:00
Shlomi Noach
e2a6b857d0 Merge branch 'master' into drop-add-same-column 2017-04-23 20:26:31 +03:00
Shlomi Noach
8a0f1413eb dropped columns are not 'shared' and no data copy attempted for such columns 2017-04-23 08:38:35 +03:00
Shlomi Noach
85c498511e parser recognizes DROP COLUMN tokens 2017-04-23 08:23:56 +03:00
Shlomi Noach
3092e9c5c0 error and nil checks on socket connection 2017-04-13 08:27:42 +03:00
Shlomi Noach
8405b0c3b8 Merge branch 'master' into close-sync-with-err 2017-03-28 08:01:18 +03:00
Shlomi Noach
7a3912da80 fixing binlog syncer double-close() 2017-03-27 08:33:06 +03:00
Shlomi Noach
c413d508cc HEAD instead of GET 2017-03-26 13:12:56 +03:00
Shlomi Noach
23ce390d69 supporting throttle-http 2017-03-26 13:10:34 +03:00
Shlomi Noach
7517238bd3 Merge branch 'master' into fix-infinite-cutover-loop 2017-03-06 21:50:26 +02:00
Shlomi Noach
91521f9357 Merge branch 'master' into old-table-name-timestamp 2017-03-05 18:10:14 +02:00
Shlomi Noach
8b2c8fe86a Merge branch 'master' into old-table-name-timestamp 2017-03-05 14:38:06 +02:00
Shlomi Noach
f11f6f978f mitigating cut-over/write race condition 2017-02-24 14:48:10 -07:00
Shlomi Noach
06c909bd10 Validating table name length 2017-02-21 17:34:49 -07:00
Shlomi Noach
8db8e127eb supporting --timestamp-old-table 2017-02-21 10:34:24 -07:00
Shlomi Noach
289ce46a2b Merge branch 'master' into fix-infinite-cutover-loop 2017-02-16 14:16:32 +02:00
Shlomi Noach
2d5f8398d6 gomysql_reader.go adapted to api changes in go-mysql 2017-02-12 13:15:53 +02:00
Shlomi Noach
1acbca105d applied gofmt 2017-02-09 09:09:35 +02:00
Shlomi Noach
109b82bb3a Merge branch 'master' into master 2017-02-09 09:04:22 +02:00
Shlomi Noach
b289041ccb Merge branch 'master' into fix-reappearing-throttled-reasons 2017-02-08 13:11:34 +02:00
Shlomi Noach
6ecf7cc0ee Auto-merged master into interactive-command-question on deployment 2017-02-08 13:03:45 +02:00
Shlomi Noach
57409c2198 added mising slaveSQLRunning value 2017-02-08 12:24:44 +02:00
Ivan Groenewold
3511aa35df improve output without --verbose option 2017-02-07 15:12:34 -03:00
Shlomi Noach
10edf3c063 Migration only starting after first replication lag metric collected 2017-02-07 12:13:19 +02:00
Shlomi Noach
91a3fd4a84 Merge branch 'master' into which-mysql-versions 2017-02-07 11:12:48 +02:00
Shlomi Noach
ed5dd7f049 Collecting and presenting MySQL versions of applier and inspector 2017-02-07 09:41:33 +02:00
Shlomi Noach
6fa32f6d54 cut-over failure on test-on-replica starts replication again 2017-02-07 09:31:52 +02:00
Shlomi Noach
9d47992d7e testing/running on replica: gets lags via SHOW SLAVE STATUS 2017-02-02 11:18:07 +02:00
Shlomi Noach
baee4f69f9 fixing phantom throttle-control-replicas lag result 2017-01-29 10:18:39 +02:00
Shlomi Noach
be1ab175c7 status presents with '# throttle-control-replicas count:' 2017-01-29 09:56:25 +02:00
Shlomi Noach
779e9fdd83 question (?) as argument in interactive commands 2017-01-29 09:25:29 +02:00
Shlomi Noach
a07a6f8cf5 fixing mediumint unsigned problem 2017-01-10 09:57:42 +02:00
Shlomi Noach
f7d2beb4d2 handling a non-DML event at the end of a dml-event sequence 2017-01-05 08:13:51 +02:00
Shlomi Noach
baaa255182 bailing out from onApplyEventStruct() 2017-01-04 12:42:21 +02:00
Shlomi Noach
645af21d03 extracted onApplyEventStruct() 2017-01-04 12:39:57 +02:00
Shlomi Noach
8d0faa55e3 explicit rollback in ApplyDMLEventQueries() 2017-01-04 08:44:04 +02:00
Shlomi Noach
220bf83a5b Merge branch 'master' into batch-apply-dml-events 2017-01-04 08:37:08 +02:00
Shlomi Noach
445c67635d batching DML writes, configurable --dml-batch-size 2017-01-03 14:31:19 +02:00
Shlomi Noach
3e28f462d8 Initial support for batching multiple DMLs when writing to ghost table 2017-01-03 13:44:52 +02:00
Shlomi Noach
9177f7553f Merge branch 'master' into heartbeat-control-replicas-unified 2016-12-30 08:07:07 +02:00
Shlomi Noach
9dd8d858a1 cli: --check-flag 2016-12-29 17:39:44 +02:00
Cyprian Nosek
2bfc672b03 Add possibility to set serverId as gh-ost argument 
Signed-off-by: Cyprian Nosek <cyprian.nosek@pearson.com>
 2016-12-28 11:49:41 +01:00
Shlomi Noach
c66356bd05 --replication-lag-query is deprecated. 2016-12-26 21:38:37 +02:00
Shlomi Noach
fc831b0548 Reading replication lag via _changelog_ table, also on control replicas 2016-12-26 21:31:35 +02:00
Shlomi Noach
eac990bc95 Merge branch 'master' into explicit-master-user-password 2016-12-15 07:58:12 +01:00
Shlomi Noach
ba2a9d9e55 support for --master-user and --master-password 2016-12-13 16:09:34 +01:00
Shlomi Noach
d12c0265b0 sleeping 500ms after 'START SLAVE' 2016-12-13 12:27:20 +01:00
Shlomi Noach
aa0932eb61 Merge branch 'master' into ask-pass 2016-12-13 11:35:52 +01:00
Shlomi Noach
7259dd6ac5 supporting --ask-pass option 2016-12-11 02:55:21 +01:00
Shlomi Noach
3fd85ee8b1 test logging cleanup 2016-12-11 02:22:56 +01:00
Shlomi Noach
1d84cb933c fix: bailing out on no PRIMARY/UNIQUE KEY 2016-12-11 02:19:19 +01:00
Shlomi Noach
35eeb56032 improved log/error messages 2016-12-05 13:41:49 +01:00
rj03hou
8f02ab0fed check the slave status when find recursive find the master, so support if the dba using reset slave instead of reset slave all. 2016-12-05 19:42:16 +08:00
Shlomi Noach
5904084e14 Merge branch 'master' into myisam-gtid 2016-12-02 15:00:44 +01:00
rj03hou
ffbd35e180 fix TableEngine correlates to the 3rd placeholder in the template string, not the 1st 2016-12-02 11:56:29 +08:00
Shlomi Noach
9b068ec222 Merge branch 'master' into myisam-gtid 2016-12-01 09:43:38 +01:00
rj03hou
a11bec1785 If the original table is MyISAM and the default engine is Innodb, and the gtid mode is on, there will be error when execute 'LOCK TABLES tbl WRITE, tbl_magic WRITE'. If make the magic cut-over table's engine same with the original table, there will not have this problem. 2016-12-01 16:04:04 +08:00
Shlomi Noach
0a707688e0 added decimal test 2016-11-29 15:47:39 +01:00
Shlomi Noach
e7cf488818 fixed parsing of quotes and of detecting rename statements 2016-11-29 15:44:38 +01:00
Shlomi Noach
5119ea4d31 added tests to verify no false positives rename-column found 2016-11-29 11:08:35 +01:00
Shlomi Noach
7126b28169 support for --skip-foreign-key-checks 2016-11-21 09:18:40 +01:00
Shlomi Noach
7ab6af8f5f never throttling inside cut-over critical section 2016-11-17 17:22:13 +01:00
Shlomi Noach
b00cae11fa retry cut-over 2016-11-17 17:10:17 +01:00
Shlomi Noach
8d987b5aaf extracted parsing of ChangelogState 2016-11-17 15:56:59 +01:00
Shlomi Noach
ef874b8551 AllEventsUpToLockProcessed uses unique signature 2016-11-17 15:50:54 +01:00
Shlomi Noach
ee447ad560 waitForEventsUpToLock timeout 
more info on AllEventsUpToLockProcessed, before and after injecting/intercepting
 2016-11-17 15:20:44 +01:00
Shlomi Noach
c2d4f624af simplified code 2016-11-03 12:14:53 +01:00
Shlomi Noach
88ffb75b8c reading and reporting replication lag before waiting on initial replication event 2016-11-02 12:48:35 +01:00
Shlomi Noach
02e9287e36 Merge branch 'master' into concurrent-rowcount-defaults-true 2016-11-01 12:22:59 +01:00
Shlomi Noach
034683f482 Merge branch 'master' into close-streamer-connection 2016-11-01 12:22:21 +01:00
Shlomi Noach
b32e1e2c16 Merge branch 'master' into throttle-no-changelog-write 2016-11-01 12:20:51 +01:00
Shlomi Noach
bb22431b83 fixed log_slave_updates check logic 2016-10-31 10:25:34 +01:00
Shlomi Noach
10850e4346 Merge branch 'master' into concurrent-rowcount-defaults-true 2016-10-28 10:14:22 +02:00
Shlomi Noach
349c9af08d Merge branch 'master' into close-streamer-connection 2016-10-28 09:30:16 +02:00
Shlomi Noach
f2bfaedcd6 Merge branch 'master' into throttle-no-changelog-write 2016-10-28 09:30:00 +02:00
Shlomi Noach
5aad45e3bc formatted code via gofmt 2016-10-27 20:38:45 +02:00
Shlomi Noach
7fa5e405d4 avoid writing heartbeat when throttle commanded by user 
when throttling on user command there really is no need for injecting heartbeat. The user commanded, therefore gh-ost complies and trusts the reasoning for throttling. What this will allow is complete quiet time. This, in turn, will allow such features as relocating via orchestrator/pseudo-gtid at time of throttling
 2016-10-27 14:51:38 +02:00
Shlomi Noach
7b63b4a275 proper cleanup of streamer connection 2016-10-27 13:52:37 +02:00
Shlomi Noach
7fe7b032e9 concurrent-rowcount defaults 'true' 2016-10-26 10:40:12 +02:00
Shlomi Noach
ac6159791d merged master, resolved conflicts 2016-10-26 09:57:59 +02:00
Shlomi Noach
b696106cb6 fixing bug introduced for charset and timezone tests 2016-10-20 15:05:47 +02:00
Shlomi Noach
bf92eec214 validating table structure on applier and migrator 
- reading column list on applier
- comparing original table on applier and migrator, expecting exact column list
- or else bailing out
 2016-10-20 11:29:30 +02:00
Shlomi Noach
82a76ae879 Merge branch 'master' into enum-sort-test 2016-10-20 10:16:02 +02:00
Akshay Chhajed
559c7895ca Modified slave update check to test tungsten flag 2016-10-19 23:31:16 +05:30
Shlomi Noach
25166e33c7 solving the enum-as-part-of-pk bug 2016-10-19 15:22:29 +02:00
Shlomi Noach
f9c15127cd simplified applier read of timezone 2016-10-14 12:56:43 +02:00
Shlomi Noach
dcd2f36c88 removed --time-zone flag 2016-10-14 12:53:23 +02:00
Shlomi Noach
9ac468c97e failing on DATETIME-to-TIMESTAMP conversion within chosen key 2016-10-14 09:33:53 +02:00
Shlomi Noach
11ccf09412 grammar 2016-10-14 09:07:30 +02:00
Shlomi Noach
e6d317b579 fixed inferred usage for --critical-load 2016-10-14 09:05:54 +02:00
Shlomi Noach
661b7ff0c1 added test to sql/types 2016-10-13 13:13:00 +02:00
Shlomi Noach
882959ba83 column is type-aware; is timezone-conversion aware 
also refactored some functions
 2016-10-13 13:12:33 +02:00
Shlomi Noach
a637ed1b50 adapted builder tests 2016-10-13 13:11:35 +02:00
Shlomi Noach
6fbf337051 builder is smarter and can use convert_tz where it notices DATETIME->TIMESTAMP conversion 2016-10-13 13:11:18 +02:00
Shlomi Noach
0981f3f7d3 fix error message placeholder 2016-10-13 13:10:16 +02:00
Shlomi Noach
84feae052f inspector notes down TIMESTAMP and DATETIME types; notices in particular the conversion from DATETIME to TIMESTAMP 2016-10-13 13:09:58 +02:00
Shlomi Noach
d80694bdac no need for Inspector timezone; it is irrelevant 2016-10-13 13:08:41 +02:00
Shlomi Noach
f988309800 no need for Inspector timezone; it is irrelevant 2016-10-13 13:08:23 +02:00
Shlomi Noach
ac304def4d applier always uses UTC 2016-10-13 13:08:02 +02:00
Shlomi Noach
184643157b Merge branch 'master' into tz-a-different-approach 2016-10-12 08:33:19 +02:00
Shlomi Noach
c11c611755 Merge pull request #260 from github/hooks-nil-check 
safe access to applier/inspector hostnames for hooks
 2016-10-12 08:16:50 +02:00
Shlomi Noach
c399119139 Merge pull request #263 from github/implied-key-fix 
assume-master-host now applies ImpliedKey
 2016-10-12 08:16:38 +02:00
Shlomi Noach
4d903d0119 Merge pull request #264 from github/discard-foreign-keys 
Discard foreign keys
 2016-10-12 08:16:27 +02:00
Shlomi Noach
5d92da4a74 Merge branch 'master' into tz-a-different-approach 2016-10-11 17:17:42 +02:00
Shlomi Noach
8bd36dcb01 Merge branch 'master' into implied-key-fix 2016-10-11 17:17:29 +02:00
Shlomi Noach
c1a6773c02 better handling of --assume-master-host 
separated logic and not even attempting to crawl topology
 2016-10-11 16:42:19 +02:00
Shlomi Noach
dbf50afbc7 reading time_zone settings for Inspector and Applier separately. 
--time-zone overrides both of them, if given
 2016-10-11 16:00:26 +02:00
Shlomi Noach
5de8a010df restoring original row_event code; tampering with applier time_zone based on global timze_zone 2016-10-10 22:33:54 +02:00
Shlomi Noach
15f4ddfd8a support for --critical-load-interval-millis 
this optional flag gives --critical-load a second chance. When configured to positive value, meeting with critical-load spawns a timer. When this timer expires a second check for critical-load is made; if still met, gh-ost bails out.
By default the interval is zero, in which case gh-ost bails out immediately on meeting critical load.
 2016-10-10 13:21:01 +02:00
Shlomi Noach
a4d566e71c counting child vs parent sie foreign keys on table, 
parent-side foreign keys cannot be discarded
 2016-10-10 12:28:56 +02:00
Shlomi Noach
6750959e1a configurable time-zone, native time parsing 2016-10-10 11:39:57 +02:00
Shlomi Noach
0c35a811f7 setting time_zoe='+00:00' on rowcopy 2016-10-08 11:06:27 +02:00
Shlomi Noach
1f65473e69 support for --discard-foreign-keys 
This flag makes migration silently and happily discard any existing foreign keys on migrated table. This is useful for intentional dropping of foreign keys, as gh-ost does not otherwise have support for foreign key migration.
At some time in the future gh-ost may support foreign key migration, at which time this flag will be removed
 2016-10-07 10:20:50 +02:00
Shlomi Noach
ef04fa49f5 assume-master-host now applied ImpliedKey 2016-10-06 12:00:34 +02:00
Shlomi Noach
72f63d3042 safe access to applier/inspector hostnames for hooks 2016-10-04 21:18:44 +02:00
Shlomi Noach
51f0c0d957 fix arg conversion in BuildDMLUpdateQuery 2016-09-27 13:08:31 +02:00
Paulo Bittencourt
2f80c9d424 Used the source column definition to fetch and convert insert values 
When processing binlog insert statements, we want to use `sharedColumns`
to decide which values to fetch and convert from the insert DML event.

We only want to  `mappedShareColumns` to define the column names in the
`replace into ...` statement.
 2016-09-14 11:28:30 -04:00
Shlomi Noach
a7627091a7 Merge branch 'master' into named-cut-over 2016-09-13 05:25:16 -07:00
Shlomi Noach
e5e0444cc6 supporting --force-named-cut-over 
- when given, user _must_ specify table name
  and of course table name must match migrated table
 2016-09-12 19:17:36 +02:00
Shlomi Noach
0a8be1dd22 excplicitly breaking on NoPrintStatusRule 2016-09-12 17:39:56 +02:00
Shlomi Noach
1c6f828091 refactored server command into server.go 
- added support for cut-over=<tablename>
- refactored more code into context
 2016-09-12 12:38:14 +02:00
Shlomi Noach
16fc19b157 rowcount progress at 100% when row-copy completes 2016-09-12 10:25:55 +02:00
Shlomi Noach
f4693274fb Extracted and generalized encoding logic to encoding.go 2016-09-08 09:37:22 +02:00
Shlomi Noach
791d963ea0 Character set recognition and manipulation 
- Identifying textual characters sets; converting into specific type when applying dml events
- Refactored `ColumnsList`: introducing `Column` type
- Refactored `unsigned` handling, as part of `Column`
- `Column` type supports `convertArg()`: converting value of argument according to column data type
- DB URI attempts `utf8mb4,utf8,latin1` charsets in that order (first one to be recognized wins)
- Local tests filter by pattern
- Local tests append table schema on failure
- Local tests do not have postpone flag file
- Added character set local tests: `utf8`, `utf8mb4`, `latin1`
 2016-09-07 14:24:11 +02:00
Shlomi Noach
88f2af8111 support for --assume-master-host, master-master/tungsten 2016-09-02 13:09:18 +02:00
Shlomi Noach
96f108d3b4 Merge pull request #221 from twotwotwo/override-applier-host 
Add -override-applier-host for use with -allow-master-master
 2016-09-02 11:32:04 +02:00
Shlomi Noach
a58e80387e More environment variables in hooks 2016-09-02 10:48:29 +02:00
Shlomi Noach
75d225353f Merge pull request #220 from Wattpad/exit-on-hook-replication-stop-failure 
Fail operation if onStopReplication hook fails
 2016-09-02 09:39:43 +02:00
randall
82110fcfcf Add -override-applier-host for use with -allow-master-master 
for configurations where writes are meant to go to one master, but gh-ost can't automatically determine which
 2016-09-01 20:29:26 -07:00
Paulo Bittencourt
e3662f2398 Fail operation if onStopReplication hook fails 2016-09-01 15:58:20 -04:00
Shlomi Noach
c562df42cd status: State and ETA decoupling 2016-09-01 10:51:40 +02:00
Shlomi Noach
904215e286 Merge pull request #204 from github/reduce-minimum-max-lag 
Reduce minimum maxLagMillisecondsThrottleThreshold to 100ms
 2016-08-31 09:29:16 +02:00
Shlomi Noach
aef56c55f7 indicating 100% when rowcopy is complete 2016-08-30 17:02:29 +02:00
Shlomi Noach
b2c71931c6 refactored all throttling code into throttler.so 2016-08-30 12:25:45 +02:00
Shlomi Noach
23357d0643 WIP: decoupling general throttling from throttle logic 2016-08-30 11:32:17 +02:00
Shlomi Noach
75b2542f26 Merge branch 'master' into reduce-minimum-max-lag 2016-08-30 09:47:33 +02:00
Shlomi Noach
1c3699cc64 fixed import path 2016-08-30 09:45:02 +02:00
Shlomi Noach
2afb86b9e4 support for millisecond throttling 
- `--max-lag-millis` is at least `100ms`
- `--heartbeat-interval-millis` introduced; defaults `500ms`, can range `100ms` - `1s`
- Control replicas lag calculated asynchronously to throttle test
  - aggressive when `max-lag-millis < 1000` and when `replication-lag-query` is given
 2016-08-30 09:41:59 +02:00
Shlomi Noach
176aabead0 Merge pull request #190 from github/hooks 
WIP: Hooks
 2016-08-30 09:01:19 +02:00
Shlomi Noach
c0936ed728 clarified privileges requirements 2016-08-29 12:39:31 +02:00
Shlomi Noach
6dfa4873c2 removed excessive argument 2016-08-29 10:44:43 +02:00
Shlomi Noach
6e5db089c8 supporting onRowCountComplete hook 2016-08-29 09:58:31 +02:00
Jonah Berquist
10b222bc7b Reduce minimum maxLagMillisecondsThrottleThreshold to 100ms 2016-08-26 16:44:40 -07:00
Shlomi Noach
c70f405d06 Merge branch 'master' into hooks 2016-08-26 08:39:02 +02:00
Shlomi Noach
4f9c1448d1 Merge pull request #200 from github/support-5-5 
fixed 5.5 binlog_row_image default value
 2016-08-26 08:29:33 +02:00
Shlomi Noach
2b595b15f2 Merge pull request #196 from github/concurrent-rowcount 
concurrent row-count
 2016-08-26 08:29:01 +02:00
Shlomi Noach
63a0e5cf2c fixed 5.5 binlog_row_image default value 2016-08-25 16:49:25 +02:00
Shlomi Noach
ef386add5c hook output and err are written to gh-ost's stderr 2016-08-25 16:08:49 +02:00
Shlomi Noach
2c9d71570f logging intented hook invocation 2016-08-25 14:42:49 +02:00
Shlomi Noach
cbdec6bd39 supporting --hooks-path and --hooks-hint 2016-08-25 13:58:36 +02:00
Shlomi Noach
ecd33969bf added onRowCountComplete(); supporting elapsedSeconds in on-status 2016-08-25 13:55:22 +02:00
Shlomi Noach
fad6743150 added hooks hint message 2016-08-25 13:54:42 +02:00
Shlomi Noach
b064174ab4 added elapsedSeconds to status hook 2016-08-25 13:54:21 +02:00
Shlomi Noach
cb1a7e2805 merged master 2016-08-25 12:32:03 +02:00
Shlomi Noach
c7d88499af Merge branch 'master' into row-copy-complete 2016-08-25 10:15:32 +02:00
Shlomi Noach
c7edd1ef84 Merge branch 'master' into concurrent-rowcount 2016-08-25 09:44:12 +02:00
Shlomi Noach
1773f338c2 keeping track of delta rows 
on concurrent count(*) this means we re-apply delta onto new estimate
 2016-08-24 12:16:34 +02:00
Shlomi Noach
553f4c8d13 concurrent row-count 2016-08-24 11:39:44 +02:00
Shlomi Noach
4c972184a8 Merge branch 'master' into sql-mode-strict 2016-08-24 09:34:00 +02:00
Shlomi Noach
56fd82a824 Merge pull request #174 from Wattpad/test-on-replica-manual-replication-control 
outstanding. Thank you!
 2016-08-24 09:12:21 +02:00
Paulo Bittencourt
6b21ade6d0 Check for --test-on-replica-skip-replica-stop in cutOver method 2016-08-23 18:34:10 -04:00
Shlomi Noach
56d09c4105 avoiding writing rows when rowcopy complete 2016-08-23 14:26:47 +02:00
Shlomi Noach
8b76d0e75b DML write sets sql_mode to STRICT ALL TABLES 2016-08-23 11:58:52 +02:00
Shlomi Noach
a1e191078a rename 'about'->'before' 2016-08-23 11:40:32 +02:00
Shlomi Noach
1c2a77ef95 hook names; added on-stop-replication hook 2016-08-23 11:35:48 +02:00
Shlomi Noach
1021a83ac0 Merge pull request #189 from dveeden/feedback_on_wait 
thank you
 2016-08-23 09:50:50 +02:00
Daniël van Eeden
d8cfd49e2c Message about waiting should be INFO not DEBUG 2016-08-23 09:41:07 +02:00
Shlomi Noach
972728cf40 added onStatus hook 2016-08-22 16:24:41 +02:00
Shlomi Noach
b63cc3e75e fix INSERT DML handling on renamed column 2016-08-22 16:00:15 +02:00
Shlomi Noach
9cf4819a98 Merge branch 'master' into fix-rename 
Wish to incorporate important time_zone fix
 2016-08-22 11:54:52 +02:00
Shlomi Noach
1376f0af23 fixed UPDATE dml on renamed column 2016-08-22 08:49:27 +02:00
Shlomi Noach
fc954562a3 Merge pull request #175 from github/apply-dml-tz 
setting time_zone on DML apply
 2016-08-22 08:01:50 +02:00
Shlomi Noach
6acbe7e3ae detecting and executing hooks 2016-08-20 08:24:20 +02:00
Paulo Bittencourt
2e43718ef3 Add --test-on-replica-skip-replica-stop flag 2016-08-19 17:34:08 -04:00
Shlomi Noach
cdf393a30e initial support for hooks 2016-08-19 14:52:49 +02:00
Shlomi Noach
d8e30fcd85 fixed sup printing heuristic 2016-08-19 09:41:25 +02:00
Shlomi Noach
9752179723 interactive command: sup 2016-08-19 09:16:17 +02:00
Shlomi Noach
6d80340e4f setting time_zone on DML apply 2016-08-19 09:06:00 +02:00
Shlomi Noach
e6a02d81e0 Merge pull request #170 from github/nice-ratio-doc-clarification 
clarifying meaning of sleep-ratio
 2016-08-19 08:27:22 +02:00
Shlomi Noach
1e88dbd04a Merge pull request #171 from github/empty-table-progress 
progress is 100% when 0/0 rows copied
 2016-08-19 08:27:04 +02:00
Shlomi Noach
4a7df07f64 Merge pull request #173 from github/accept-mysql-prompt-config 
relaxed config scanner mode
 2016-08-19 08:26:13 +02:00
Shlomi Noach
41d43933c8 Merge pull request #172 from github/transactional-apply-dml 
DML write wrapped in transaction
 2016-08-19 08:25:52 +02:00
Paulo Bittencourt
a62f9e0754 Add --test-on-replica-manual-replication-control flag 
This will wait indefinitely for the replication status to change.
This allows us to run test schema changes in RDS without needing
custom RDS commands in gh-ost.
 2016-08-18 11:53:25 -04:00
Shlomi Noach
00369d7e5d relaxed config scanner mode 
- does not fail on MySQL 'prompt' config
 2016-08-18 13:58:38 +02:00
Shlomi Noach
88eb2d6ee1 Merge pull request #130 from dveeden/foreign_key_info 
Give more info about foreign keys
 2016-08-18 13:40:31 +02:00
Shlomi Noach
75e0d12302 simplified error logic; fixed incorrect RowsEstimate handling on error 2016-08-18 13:38:23 +02:00
Shlomi Noach
74593ec010 DML write wrapped in transaction 
- solving the golang problem: 'sql: converting Exec argument #2's type: uint64 values with high bit set are not supported'
 2016-08-18 13:31:53 +02:00
Shlomi Noach
7e9f578e12 progress is 100% when 0/0 rows copied 2016-08-18 13:20:09 +02:00
Shlomi Noach
5dbd2e1c85 clarifying meaning of sleep-ratio 2016-08-18 13:13:51 +02:00
Shlomi Noach
d9ae2f3942 Merge pull request #156 from github/avoid-restarting-replication 
accepting `--assume-rbr`, remove `SUPER` requirement
 2016-08-18 10:11:35 +02:00
Daniël van Eeden
d3422bd19a Give more info about foreign keys 
This helps when the table itself doesn't have foreign keys defined but if there are other tables with foreign keys pointing to the table on which gh-ost runs.

This gives INFO messages for each FK. Note that it now informs the user about `payment` being involved.
```
$ ./gh-ost -database sakila -table rental -alter 'ADD COLUMN ghost_test_001 tinyint DEFAULT NULL' -port 19590 -user msandbox -password msandbox -verbose
2016-08-03 10:18:45 INFO starting gh-ost 1.0.8
2016-08-03 10:18:45 INFO Migrating `sakila`.`rental`
2016-08-03 10:18:45 INFO connection validated on 127.0.0.1:19590
2016-08-03 10:18:45 INFO User has ALL privileges
2016-08-03 10:18:45 INFO binary logs validated on 127.0.0.1:19590
2016-08-03 10:18:45 INFO Restarting replication on 127.0.0.1:19590 to make sure binlog settings apply to replication thread
2016-08-03 10:18:46 INFO Table found. Engine=InnoDB
2016-08-03 10:18:47 INFO Found foreign key on `sakila`.`payment` related to `sakila`.`rental`
2016-08-03 10:18:47 INFO Found foreign key on `sakila`.`rental` related to `sakila`.`rental`
2016-08-03 10:18:47 INFO Found foreign key on `sakila`.`rental` related to `sakila`.`rental`
2016-08-03 10:18:47 INFO Found foreign key on `sakila`.`rental` related to `sakila`.`rental`
2016-08-03 10:18:47 ERROR Found 4 foreign keys related to `sakila`.`rental`. Foreign keys are not supported. Bailing out
2016-08-03 10:18:47 FATAL 2016-08-03 10:18:47 ERROR Found 4 foreign keys related to `sakila`.`rental`. Foreign keys are not supported. Bailing out
```

Related Issue: https://github.com/github/gh-ost/issues/129
 2016-08-18 10:07:32 +02:00
Shlomi Noach
3a0ee9b4a5 clarified commented transactional apply 2016-08-17 10:50:41 +02:00
Shlomi Noach
16d76aa299 builder fixes arg type from signed to unsigned based on UnsingedFlags 2016-08-17 06:52:23 +02:00
Shlomi Noach
29d20316ba inspector applies unsigned flags to all migration related columns 2016-08-17 06:51:58 +02:00
Shlomi Noach
f00a9814e6 legacy comments cleanup 2016-08-17 06:51:06 +02:00
Shlomi Noach
4c8edf6372 elaborate error message on applying event data: printing out the error, query and args 2016-08-17 06:50:40 +02:00
Shlomi Noach
7a0e2cfe3f adding UnsignedFlags to ColumnsList 2016-08-17 06:49:20 +02:00
Shlomi Noach
596dce5993 elaborate output on error in apply dml 2016-08-15 15:23:30 +02:00
Shlomi Noach
ac0b788153 rename trust-rbr to assume-rbr 2016-08-15 11:05:51 +02:00
Shlomi Noach
01d48e667c Merge pull request #143 from EasyPost/issue_142 
Thank you
 2016-08-15 10:53:36 +02:00
Shlomi Noach
1995be2b3f accepting --trust-rbr 
- avoiding need to restart replication
- in turn avoiding need for SUPER
 2016-08-12 14:26:58 +02:00
Shlomi Noach
8bf07c506f Merge pull request #147 from github/cleanup-socket-file 
Cleanup socket file
 2016-08-12 11:26:37 +02:00
Shlomi Noach
51111e0aa7 Merge pull request #155 from github/relaxed-log-slave-updates 
relaxed check for log_slave_updates
 2016-08-12 11:25:18 +02:00
Shlomi Noach
a46022f727 localized function name 2016-08-11 17:37:50 +02:00
Shlomi Noach
66ff5964ed relaxed check for log_slave_updates 2016-08-11 14:49:14 +02:00
Shlomi Noach
95b4f9e440 testing for trigger existence 2016-08-11 14:10:35 +02:00
Shlomi Noach
a1f9c47102 Merge pull request #137 from dveeden/grants_info 
Thank you
 2016-08-11 09:07:48 +02:00
Shlomi Noach
35ff3f5b7b added log message 2016-08-11 09:03:39 +02:00
Shlomi Noach
dd1ef29dac cleaning up socket file 2016-08-11 09:01:14 +02:00
Shlomi Noach
1418395e99 Merge pull request #124 from dgryski/time-since 
Thank you!
 2016-08-11 08:50:06 +02:00
James Brown
b2fee9a037 detect IPv6 literals and wrap them in square brackets in GetDBUri 2016-08-05 16:54:26 -07:00
Daniël van Eeden
7786356a45 Fix typo: s/auto_incrmenet/auto_increment/ 2016-08-05 17:46:38 +02:00
Daniël van Eeden
95ba07e01b More user friendly error reporting for privileges 
Related Issue: https://github.com/github/gh-ost/issues/136

New output:
```
2016-08-05 11:39:07 DEBUG Privileges: Super: true, REPLICATION SLAVE: false, ALL on *.*: false, ALL on `sakila`.*: true
2016-08-05 11:39:07 ERROR User has insufficient privileges for migration. Needed: SUPER, REPLICATION SLAVE and ALL on `sakila`.*
2016-08-05 11:39:07 FATAL 2016-08-05 11:39:07 ERROR User has insufficient privileges for migration. Needed: SUPER, REPLICATION SLAVE and ALL on `sakila`.*
```
 2016-08-05 11:41:36 +02:00
Seth Krasnianski
a94f64b42c [go cmd] Correct spelling for main() comment 2016-08-02 09:22:44 -04:00
Damian Gryski
e02a49449e all: use time.Since() instead of time.Now().Sub 
Patch created with:
    gofmt -w -r 'time.Now().Sub(a) -> time.Since(a)' .
 2016-08-02 08:38:56 -04:00
Shlomi Noach
46bbea2a32 ETA counting rows, fixed copy time on count 2016-07-29 10:40:23 +02:00
Shlomi Noach
25ce8b0758 status hint parameters using normalized names 2016-07-29 09:20:00 +02:00
Shlomi Noach
edacb8f959 Merge pull request #116 from github/nice-ratio-float 
nice-ratio is now float64
 2016-07-29 07:16:33 +02:00
Shlomi Noach
be8a023350 nice-ratio is now float64 2016-07-28 14:37:17 +02:00
Shlomi Noach
b99ce969c7 serving socket before counting table rows 2016-07-28 13:01:26 +02:00
Shlomi Noach
b548a6a172 adding human friendly hint re: throttling and binary logs 2016-07-27 10:45:22 +02:00
Shlomi Noach
dbcc0e09c7 status hint shows [set] next to existing flag files 2016-07-27 10:36:24 +02:00
Shlomi Noach
e900dae2e9 More informative information upon control-replicas lagging 2016-07-27 09:59:46 +02:00
Shlomi Noach
b53ee24a1f dynamic replication-lag-query 2016-07-26 14:14:25 +02:00
Shlomi Noach
5d23b72955 Merge pull request #107 from github/throttle-control-replicas 
fix to throttle-control-replicas check
 2016-07-26 12:13:22 +02:00
Shlomi Noach
7a70c24503 replica-migration cleanup; updating allEventsUpToLockProcessedInjectedFlag 2016-07-26 12:06:20 +02:00
Shlomi Noach
6dbf5c31a2 resolved conflict 2016-07-26 11:57:01 +02:00
Shlomi Noach
034ea7646a fix to throttle-control-replicas check 2016-07-26 11:51:24 +02:00
Shlomi Noach
a75912d1a4 Merge pull request #105 from github/credentials-env-variables 
config file supports environment variables
 2016-07-26 09:28:23 +02:00
Shlomi Noach
4774b67ffd config file supports environment variables 2016-07-25 15:46:37 +02:00
Shlomi Noach
1d77425fbe capped streamer retries 2016-07-25 15:17:30 +02:00
Shlomi Noach
74804559c8 supporting --initially-drop-socket-file 
- by default gh-ost will not delete an existing socket file
  and thus, will fail running if socket file exists. This is the desired behavior.
- The flag --initially-drop-socket-file indicates we take responsibility and wish gh-ost to delete this file on startup
 2016-07-22 17:34:18 +02:00
Shlomi Noach
ef59a866d8 Removed legacy 'safe cut-over' 
Now that we have the atomic cut-over, the former is redundant
 2016-07-16 08:12:19 -06:00
Shlomi Noach
8e46b4ceea max-lag-millis is dynamicly controllable 2016-07-13 09:44:00 +02:00
Shlomi Noach
8217536898 supporting --cut-over-lock-timeout-seconds 2016-07-08 10:14:58 +02:00
Shlomi Noach
c116d84acb added nice-ratio 2016-07-04 14:29:09 +02:00
Shlomi Noach
37e3c94c87 supporting 'unpostpone' command 2016-07-01 10:59:09 +02:00
Shlomi Noach
0191b2897d an atomic cut-over implementation, as per issue #82 2016-06-27 11:08:06 +02:00
Shlomi Noach
4f299f320e noop more verbose 2016-06-27 08:49:26 +02:00
Shlomi Noach
e0de69b028 a noop operation dumps SHOW CREATE TABLE 2016-06-22 12:39:13 +02:00
Shlomi Noach
5b20122957 on noop operation, drop ghost table at end 2016-06-22 10:48:17 +02:00
Shlomi Noach
690e046c51 adding --allow-master-master 2016-06-22 10:38:13 +02:00
Shlomi Noach
96e8419a35 Solved cut-over stall; change of table names 
- Cutover would stall after `lock tables` wait-timeout due do waiting on a channel that would never be written to. This has been identified, reproduced, fixed, confirmed.
- Change of table names. Heres the story:
  - Because were testing this even while `pt-online-schema-change` is being used in production, the `_tbl_old` naming convention makes for a collision.
  - "old" table name is now `_tbl_del`, "del" standing for "delete"
  - ghost table name is now `_tbl_gho`
  - when issuing `--test-on-replica`, we keep the ghost table around, and were also briefly renaming original table to "old". Well this collides with a potentially existing "old" table on master (one that hasnt been dropped yet).
  `--test-on-replica` uses `_tbl_ght` (ghost-test)
  - similar problem with `--execute-on-replica`, and in this case the table doesnt stick around; calling it `_tbl_ghr` (ghost-replica)
  - changelog table is now `_tbl_ghc` (ghost-changelog)
  - To clarify, I dont want to go down the path of creating "old" tables with 2 or 3 or 4 or 5 or infinite leading underscored. I think this is very confusing and actually not operations friendly. Its OK that the migration will fail saying "hey, you ALREADY have an old table here, why dont you take care of it first", rather than create _yet_another_ `____tbl_old` table. Were always confused on which table it actually is that gets migrated, which is safe to `drop`, etc.
- just after rowcopy completing, just before cutover, during cutover: marking as point in time _of interest_ so as to increase logging frequency.
 2016-06-21 12:56:01 +02:00
Shlomi Noach
cd6b3c5e9e not throttling during cut-over operation 2016-06-21 09:21:58 +02:00
Shlomi Noach
80fcc05eb5 supporting interactive command throttle-control-replicas 2016-06-20 12:09:04 +02:00
Shlomi Noach
f0b012b238 support for 'panic' interactive command 2016-06-20 06:38:29 +02:00
Shlomi Noach
62b8a897e3 Retries, better visibility, documentation 
- Rowcopy time is bounded by copy end-time
- Retries are configurable via `--default-retries` (default: `60`)
- `migrator` notes the hostname
- `applier` and `inspector` note `impliedKey` (`@@hostname` and `@@port`)
- Added lots of code comments
- Adding documentation for "triggerless design"
 2016-06-19 17:55:37 +02:00
Shlomi Noach
23cb8ea7e9 Throttling & critical load 
- Added `--throttle-query` param (when returns > 0, throttling applies)
- Added `--critical-load`, similar to `--max-load` but implies panic and quit
- Recoded *-load as `LoadMap`
- More info on *-load throttle/panic
- `printStatus()` now gets printing heuristic. Always shows up on interactive `"status"`
- Fixed `change column` (aka rename) handling with quotes
- Removed legacy `mysqlbinlog` parser code
- Added tests
 2016-06-18 21:12:07 +02:00
Shlomi Noach
d38ff68a15 minor formatting 2016-06-17 11:41:10 +02:00
Shlomi Noach
94f311ec7b supporting --panic-flag-file; when it exists - app panics and exits without cleanup 2016-06-17 11:40:08 +02:00
Shlomi Noach
836d0fe119 Supporting column rename 
- Parsing `alter` statement to catch `change old_name new_name ...` statements
- Auto deducing renamed columns
- When suspecting renamed columns, requesting explicit `--approve-renamed-columns` or `--skip-renamed-columns`
- updated tests
 2016-06-17 08:03:18 +02:00
Shlomi Noach
3e83202b97 more elaborate check that user has privileges 2016-06-16 16:06:26 +02:00
Shlomi Noach
7d0ec9c9dc added --migrate-on-replica flag; runs complete migration on replica 2016-06-15 12:18:59 +02:00
Shlomi Noach
85d6883e69 printing migration status on waitForEventsUpToLock() 2016-06-15 10:13:06 +02:00
Shlomi Noach
96bc3804eb test-on-replica stops replication completely 2016-06-14 12:50:07 +02:00
Shlomi Noach
97adbf1ff8 - --cut-over no longer mandatory; default to safe 
- Removed `CutOverVoluntaryLock` and associated code
- Removed `CutOverUdfWait`
- `RenameTablesRollback()` first attempts an atomic swap
 2016-06-14 09:01:06 +02:00
Shlomi Noach
cb1c61ac47 - --cut-over no longer mandatory; default to safe 
- Removed `CutOverVoluntaryLock` and associated code
- Removed `CutOverUdfWait`
- `RenameTablesRollback()` first attempts an atomic swap
 2016-06-14 09:00:56 +02:00
Shlomi Noach
8292f5608f Safe cut-over 
- Supporting multi-step, safe cut-over phase, where queries are blocked throughout the phase, and worst case scenario is table outage (no data corruption)
- Self-rollsback in case of failure (restored original table)
 2016-06-14 08:35:07 +02:00
Shlomi Noach
e4ed801df5 noting posponing status 2016-06-13 18:36:29 +02:00
Shlomi Noach
b8c7e046a1 test-on-replica to invoke cut-over swap 2016-06-10 11:15:11 +02:00
Shlomi Noach
087d1dd64d suuporting dynamic reconfiguration of max-load 2016-06-09 11:25:01 +02:00
Shlomi Noach
2cdc72bd1c fixed nil TCP listener when TCP undfined 2016-06-07 14:24:30 +02:00
Shlomi Noach
a6c21dcdb0 - --postpone-swap-tables-flag-file renamed to --postpone-cut-over-flag-file 
- More `README` documentation
- Added "throttle" documentation
 2016-06-07 14:05:25 +02:00
Shlomi Noach
fc00cb2289 adding interactive user commands 2016-06-07 11:59:17 +02:00
Shlomi Noach
bbd19abc9a - requiring --cut-over argument to be two-step|voluntary-lock (will add udf-wait once it is ready) 
The idea is that the user is forced to specify the cut-over type they wish to use, given that each type has some drawbacks.
- More data in status hint
- `select count(*)` is deferred till after we validate migration is valid. Also, it is skipped on `--noop`
 2016-06-06 12:33:05 +02:00
Shlomi Noach
42ae3e37f5 dropping _osc (changelog) table at end of operation; also better status hint at end of operation 2016-06-01 10:40:49 +02:00
Shlomi Noach
2df94f9c51 printing courtesy reminder once per 10 minutes 2016-05-31 21:12:39 +02:00
Shlomi Noach
9519a66825 added courtesy-reminder 2016-05-26 14:25:32 +02:00
Shlomi Noach
583d6d3147 accepting SIGHUP. Reloads configuration and marks as point of interest 2016-05-25 12:27:58 +02:00
Shlomi Noach
e7239091d7 Merge pull request #45 from github/print-status-point-of-interest 
support for marking point-of-interest in migration
 2016-05-24 08:48:44 +02:00
Shlomi Noach
20f000833f support for marking point-of-interest in migration 2016-05-23 14:58:53 +02:00
Shlomi Noach
896f560dce after timeout: reconnecting as new replica; skipping queries correctly 2016-05-23 11:12:59 +02:00
Shlomi Noach
5375aa4f69 - Removed use of master_pos_wait(). It was unneccessary in the first place and introduced new problems. 
- Supporting `--allow-nullable-unique-key`
  - Tool will bail out if chosen key has nullable columns and the above is not provided
- Fixed `OriginalBinlogRowImage` comaprison (lower/upper case issue)
- Introduced reasonable streamer reconnect sleep time
 2016-05-20 12:52:14 +02:00
Shlomi Noach
9b54d0208f - Handling gomysql.replication connection timeouts: reconnecting on last known position 
- `printStatus()` takes ETA into account
- More info around `master_pos_wait()`
 2016-05-19 15:11:36 +02:00
Shlomi Noach
ec34a5ef75 master_pos_wait is now OK to return NULL. We only care if it returns with -1 2016-05-18 15:08:47 +02:00
Shlomi Noach
9f56a84b57 Fixing single-row table migration 
- `BuildUniqueKeyRangeEndPreparedQuery` supports `includeRangeStartValues` argument
- `applier` sends `this.migrationContext.GetIteration() == 0` as argument
 2016-05-18 14:53:09 +02:00
Shlomi Noach
45371d9374 Merge pull request #36 from github/master-pos-wait-fix 
some messagages are now Info instead of Debug
 2016-05-18 12:21:22 +02:00
Shlomi Noach
df0a7513f5 - user/password provided in CLI override those in config file 
- user no longer defaults to .
- config is now part of Context, and is protected by mutex
 2016-05-17 15:35:44 +02:00
Shlomi Noach
879b2b425e - Support for --postpone-swap-tables-flag-file: while this file exists, final table swap does not take place, and the ghost table keeps being synchronized 
- Fixed version printing
- `rowCopyCompleteFlag` is a hint that allows us to escape the infinite loop of rowcopy once we are sure we have reached the end
 2016-05-17 14:40:37 +02:00
Shlomi Noach
065d9c40ec some messagages are now Info instead of Debug 2016-05-17 11:57:43 +02:00
Shlomi Noach
41b0a4f317 supporting --version 2016-05-17 11:51:21 +02:00
Shlomi Noach
21f6ae9dca renaming to gh-ost 2016-05-16 11:10:12 +02:00
Shlomi Noach
9d055dbda7 renaming to gh-ost 2016-05-16 11:09:17 +02:00
Shlomi Noach
1e10f1f29e Solved various race conditions: 
- Operation would terminate after events lock noticed but before applying all events: race condition where the event would be captured asynchronously. The event is now handled sequentially with the DML events, hence now safe.
- Multiple rowcopy operations would still write to `rowCopyComplete` channel. This is still the case, but now we only wait for the first and then just flush (read and discard) any others, to avoid blocking
- Events DML listener is only added after table creation: the problem was that with very busy tables, the events func buffer would fill up, and the "tables-created" event would be blocked.
- `waitForEventsUpToLock()` unifies the waiting on all variants of complete-migration
- With `--test-on-replica`, now stopping replication "nicely", using `master_pos_wait()`
- With `--test-on-replica`, not throttling on replication after replication is stopped (duh)
- More debug output
 2016-05-16 11:03:15 +02:00
Shlomi Noach
134bf385fd initial, simple solution to our-of-order applying of DML events 2016-05-05 17:14:55 +03:00
Shlomi Noach
6528010742 Adding ETA starting at 2% progress 2016-05-05 09:18:19 +03:00
Shlomi Noach
800c1109b0 fixed statistics query: getting the correct column names by unique key 2016-05-04 09:50:00 +03:00
Shlomi Noach
74d8b06db1 exact-rowcount implices updating number of rows as we make progress 2016-05-04 08:23:34 +03:00
Shlomi Noach
36905d82e3 - supporting --initially-drop-old-table 
- supporting `--initially-drop-ghost-table`
- validating existence of `old` and `ghost` before beginning operation
 2016-05-03 12:55:17 +03:00
Shlomi Noach
627e412b6b fixed password assignment 2016-05-03 11:56:53 +03:00
Shlomi Noach
86fd2b617a initial support for config file 2016-05-03 10:28:48 +03:00
Shlomi Noach
07063a4181 - added throttle-control-replicas flag, a list of control replicas 
- when `--test-on-replica`, the tested replica is implicitly a control replica
- added `replication-lag-query`, an alternate query to `SHOW SLAVE STATUS` to get replication lag
- throttling takes both the above into consideration
 2016-05-01 21:36:36 +03:00
Shlomi Noach
421ab0fc83 woohoo, logic complete 
- Introduced `SwapTablesTimeoutSeconds`; `RENAME` is limited by this timeout
- If `RENAME` fails (due to the above), we throttle and retry
- `SwapTablesAtomic()` sets `lock_wait_timeout` and notifies with connection id
- `GrabVoluntaryLock()` intentionally grabs (and later releases) voluntary lock. It notifies when it is taken and awaits instructions as for when it could be released.
- `IssueBlockingQueryOnVoluntaryLock()` does what it says. It notifies with its connection_id so that it can be easily traced
- `stopWritesAndCompleteMigrationOnMasterViaLock()` does the thang. Oh dear this was agonizing and the code is a pain to look at, though under the limitations I do believe it is as clean as I could hope for.
 2016-04-22 19:46:34 -07:00
Shlomi Noach
1ed1b0d156 - quick-and-bumpy-swap-tables uses quicker swap tables, at the expense of a period where the table does not exist (non atomic renames) 
- refactored lock-and-swap code, in preparation for atomic swap
 2016-04-22 13:41:20 -07:00
Shlomi Noach
54c6d059b5 - quick-and-bumpy-swap-tables uses quicker swap tables, at the expense of a period where the table does not exist (non atomic renames) 
- refactored lock-and-swap code, in preparation for atomic swap
 2016-04-22 13:18:56 -07:00
Shlomi Noach
3c85298b77 - Better, fewer NOOP checks around the code 
- Keeping track of `TotalDMLEventsApplied`
 2016-04-19 04:25:32 -07:00
Shlomi Noach
4efbfd6e0f Merge pull request #19 from github/oops-leftovers 
oops, leftover file
 2016-04-18 10:59:58 -07:00
Shlomi Noach
9dce88e6c0 oops, leftover file 2016-04-18 10:59:34 -07:00
Shlomi Noach
eeffa701d6 - Added ok-to-drop-table flag 
- Added `switch-to-rbr` flag; applying binlog format change if needed
- Using dedicated db instance for locking & renaming on applier (must be used from within same connection)
- Heartbeat now uses `time.RFC3339Nano`
- Swap tables works! Caveat: short table outage
- `--test-on-replica` works!
- retries: using `panicAbort`: from any goroutine, regardless of context, it is possible to terminate the operation
- Reintroduced changelog events listener on streamer. This is the correct implementation.
 2016-04-18 10:57:18 -07:00
Shlomi Noach
a4ee80df13 - Building and applying queries from binlog event data! 
- `INSERT`, `DELETE`, `UPDATE` statements
- support for `--noop`
- initial support for `--test-on-replica`. Verifying against `--allow-on-master`
- Changelog events no longer read from binlog stream, because reading it may be throttled, and we have to be able to keep reading the heartbeat and state events.
  They are now being read directly from table, mapping already-seen-events to avoid confusion
  Changlelog listener pools table in 2*frequency of heartbeat injection
 2016-04-14 13:37:56 +02:00
Shlomi Noach
0d25d11b40 added types_test 2016-04-11 19:06:47 +02:00
Shlomi Noach
04525887f3 - Throttling-check is now an async routine running once per second 
- Throttling variables protected by mutex
- Added `--throttle-additional-flag-file`: `operation pauses when this file exists; hint: keep default, use for throttling multiple gh-osc operations`
- ColumnList is not a `struct` which contains ordinal mapping
- More implicit write changelog + audit changelog
- builder now builds `DELETE` and `INSERT` queries from data it will eventually get from DML event
- Sanity check for binlog_row_image
- Restarting replication to be sure binlog settings apply
- Prepare for accepting `SIGHUP` (reloading configuration)
 2016-04-11 17:27:16 +02:00
Shlomi Noach
80163b35b6 extracted on-throttle functions outside loop 2016-04-08 14:44:36 +02:00
Shlomi Noach
a1a34b8150 ongoing development: 
- accepts --max-load
- accepts multiple conditions in --max-load
- throttle includes reason
- chunk-size sanity check
- change log state writes both in appending (history) mode and in replacing (current) mode
- more atomic checks
- inspecting ghost table columns, unique key
- comparing unique keys between tables; sanity
- intersecting columns between tables
- prettify status
- refactored throttle() and retries()
 2016-04-08 14:35:06 +02:00
Shlomi Noach
75f68c0752 - row copy and row events are now handled by a single routine which prioritizes events over rowcopy 
- Supporting `--throttle-file-flag`
- Printing status
- Supporting transactional table syntax
- code cleanup; refactoring
- proper use of atomic where required
- iterations are in changelog (erm... maybe too much)
- `LOCK TABLES`, `UNLOCK TABLES` working
 2016-04-08 10:34:44 +02:00
Shlomi Noach
0e7b23e6fe - Creating an populating Changelog table 
- Using heartbeat
- Throttling works based on heartbeat
- Refactored binlog_reader stuff. Now streaming events (into golang channel, which makes for nice buffering and throttling)
- Binlog table listeners work
- More Migrator logic; existing logic for waiting on `state` events (e.g. `TablesCreatedState`)
 2016-04-07 15:57:12 +02:00
Shlomi Noach
4dd5a93ed7 - More work on go-mysql reader: now properly notes binlog coordinates as it makes progress 
- Properly creating multiple BinlogEntry-ies when reading multi statement binlog event (Still todo: transactions)
 2016-04-06 18:44:54 +02:00
Shlomi Noach
5deff2adb6 - working POC of row-copy iteration cycle 
- initial work on table columns
- initial work on events streamer
 2016-04-06 13:05:58 +02:00
Shlomi Noach
f771016bd5 . 2016-04-06 13:05:21 +02:00
Shlomi Noach
d8fefb3d6f exploded args on range query building; iteration works 2016-04-05 19:50:49 +02:00
Shlomi Noach
3583ab5dc5 beginning support for ranges and iteration. Still WIP 2016-04-05 09:14:22 +02:00
Shlomi Noach
ea0906f4e5 reading table (range) min/max values, right now according to hardcoded unique key 2016-04-04 18:19:46 +02:00
Shlomi Noach
937491674c adding applier, instance_key, instance_key_map 2016-04-04 15:30:49 +02:00
Shlomi Noach
cf87d16044 detecting master (includes sanity checks). Introducing Applier. Creating and altering ghost table 2016-04-04 15:29:02 +02:00
Shlomi Noach
c75cd998fb a bunch of 'inspector' initial tests on the replica 2016-04-04 12:27:51 +02:00
Shlomi Noach
1b8c5ff286 preparing for db access 2016-04-01 16:05:54 +02:00
Shlomi Noach
f5b276415a initial work on context 2016-04-01 16:05:44 +02:00
Shlomi Noach
b461c320cc added BuildUniqueKeyRangeEndPreparedQuery() 2016-04-01 16:05:17 +02:00
Shlomi Noach
5e5b868c5c Merge pull request #9 from github/build-range-prepared 
building prepared queries (with positional params)
 2016-04-01 13:56:33 +02:00
Shlomi Noach
41c3d15963 building prepared queries (with positional params) 2016-04-01 13:55:51 +02:00
Shlomi Noach
9b368a7720 Merge pull request #8 from github/sql-queries-manipulations 
merging so I can use this on other branches
 2016-04-01 13:54:20 +02:00
Shlomi Noach
39ebc75c43 initial work on sql query building 2016-04-01 13:36:56 +02:00
Shlomi Noach
8f3d13e071 still experimenting, yes? go-mysql binlog parser looks good 2016-03-30 15:43:40 +02:00
Shlomi Noach
30a472f741 begin work on positional columns. Incomplete 2016-03-24 22:27:39 +01:00
Shlomi Noach
96a8fd50c3 minor refactoring; fixed license comments 2016-03-24 15:11:56 +01:00
Shlomi Noach
fb04eb232f simplified appendBinlogEntry() 2016-03-24 14:52:49 +01:00
Shlomi Noach
8aa6a9750c adjusted automaton to accept multi-statement entries; added tests 2016-03-24 14:25:52 +01:00
Shlomi Noach
1768b55b3b mysqlbinlog_reader is now a simple state machine 2016-03-23 15:25:45 +01:00
Shlomi Noach
e36bb51b69 initial seems-to-be-working parser for binary logs, which reads log pos, end log pos, statement type, schema and table. Performs some validations along the way. Reads in chunks of up to 32M per chunk 2016-03-23 12:40:17 +01:00
Shlomi Noach
09046c8f6e initial binlog package 2016-03-22 15:12:51 +01:00
Shlomi Noach
bb1c6dae30 adding initial bunch of code 2016-03-21 15:57:01 +01:00