Shlomi Noach
79e4f1cdbe
detect range end based on OFFSET
2017-08-21 08:12:41 +03:00
Nikhil Mathew
a1473dfb58
Stop infinite goroutines
2017-08-08 15:31:25 -07:00
Nikhil Mathew
7412f42371
Remove no longer needed acceptSignals func
2017-08-08 13:44:53 -07:00
Nikhil Mathew
b399533d4f
Simplify GoMySQLReader
2017-08-08 13:41:46 -07:00
Nikhil Mathew
982b8eede9
Refactor global migration context
2017-08-08 13:36:54 -07:00
Shlomi Noach
de0ebcbbb0
documentation
2017-07-19 16:48:22 +03:00
Shlomi Noach
6da0b8af3d
Dynamic DML batch size; apadting buffer size to match
2017-07-19 16:44:18 +03:00
Shlomi Noach
4a3a5a1ce1
Merge branch 'master' into case-insensitive-columns
2017-07-16 06:51:39 +03:00
Shlomi Noach
effae08fbf
supporting change of column name-case
2017-07-12 12:33:32 +03:00
Shlomi Noach
e6dd2c90d2
responding to errors in rowcopy
2017-07-11 13:39:11 +03:00
Shlomi Noach
a8932ee1b8
freno-aware http-throttler
2017-07-04 10:28:09 +03:00
Shlomi Noach
36bcc031ce
Merge branch 'master' into fix-variables-spelling
2017-06-29 20:53:49 +03:00
Shlomi Noach
cb3c7f3e20
Merge branch 'master' into touch-postpone-flag-file
2017-06-11 08:57:45 +03:00
Shlomi Noach
8da0f60582
fixed critical-load check for hibernation
2017-05-24 10:53:00 +03:00
Shlomi Noach
ad47f7c147
throttling just prior to leaving hibernation, so as to allow re-throttle checks to apply
2017-05-24 10:42:47 +03:00
Shlomi Noach
3955a6d67f
hibernate on critical-load
2017-05-24 08:32:13 +03:00
Shlomi Noach
aca288e88d
-postpone-cut-over-flag-file implies touching indicated file
2017-05-07 14:58:18 +03:00
Shlomi Noach
eaa3489fe0
Merge branch 'master' into server-report-coordinates
2017-05-01 07:50:13 +03:00
Shlomi Noach
79cee99f57
support for 'coordinates' command
2017-04-28 15:50:51 -07:00
Jacob Bednarz
0243e76b33
s/applyEnvironmentVairables/applyEnvironmentVariables
2017-04-24 07:45:21 +10:00
Shlomi Noach
e2a6b857d0
Merge branch 'master' into drop-add-same-column
2017-04-23 20:26:31 +03:00
Shlomi Noach
8a0f1413eb
dropped columns are not 'shared' and no data copy attempted for such columns
2017-04-23 08:38:35 +03:00
Shlomi Noach
3092e9c5c0
error and nil checks on socket connection
2017-04-13 08:27:42 +03:00
Shlomi Noach
c413d508cc
HEAD instead of GET
2017-03-26 13:12:56 +03:00
Shlomi Noach
23ce390d69
supporting throttle-http
2017-03-26 13:10:34 +03:00
Shlomi Noach
7517238bd3
Merge branch 'master' into fix-infinite-cutover-loop
2017-03-06 21:50:26 +02:00
Shlomi Noach
91521f9357
Merge branch 'master' into old-table-name-timestamp
2017-03-05 18:10:14 +02:00
Shlomi Noach
f11f6f978f
mitigating cut-over/write race condition
2017-02-24 14:48:10 -07:00
Shlomi Noach
06c909bd10
Validating table name length
2017-02-21 17:34:49 -07:00
Shlomi Noach
289ce46a2b
Merge branch 'master' into fix-infinite-cutover-loop
2017-02-16 14:16:32 +02:00
Shlomi Noach
b289041ccb
Merge branch 'master' into fix-reappearing-throttled-reasons
2017-02-08 13:11:34 +02:00
Shlomi Noach
6ecf7cc0ee
Auto-merged master into interactive-command-question on deployment
2017-02-08 13:03:45 +02:00
Shlomi Noach
10edf3c063
Migration only starting after first replication lag metric collected
2017-02-07 12:13:19 +02:00
Shlomi Noach
91a3fd4a84
Merge branch 'master' into which-mysql-versions
2017-02-07 11:12:48 +02:00
Shlomi Noach
ed5dd7f049
Collecting and presenting MySQL versions of applier and inspector
2017-02-07 09:41:33 +02:00
Shlomi Noach
6fa32f6d54
cut-over failure on test-on-replica starts replication again
2017-02-07 09:31:52 +02:00
Shlomi Noach
9d47992d7e
testing/running on replica: gets lags via SHOW SLAVE STATUS
2017-02-02 11:18:07 +02:00
Shlomi Noach
baee4f69f9
fixing phantom throttle-control-replicas lag result
2017-01-29 10:18:39 +02:00
Shlomi Noach
be1ab175c7
status presents with '# throttle-control-replicas count:'
2017-01-29 09:56:25 +02:00
Shlomi Noach
779e9fdd83
question (?) as argument in interactive commands
2017-01-29 09:25:29 +02:00
Shlomi Noach
a07a6f8cf5
fixing mediumint unsigned problem
2017-01-10 09:57:42 +02:00
Shlomi Noach
f7d2beb4d2
handling a non-DML event at the end of a dml-event sequence
2017-01-05 08:13:51 +02:00
Shlomi Noach
baaa255182
bailing out from onApplyEventStruct()
2017-01-04 12:42:21 +02:00
Shlomi Noach
645af21d03
extracted onApplyEventStruct()
2017-01-04 12:39:57 +02:00
Shlomi Noach
8d0faa55e3
explicit rollback in ApplyDMLEventQueries()
2017-01-04 08:44:04 +02:00
Shlomi Noach
220bf83a5b
Merge branch 'master' into batch-apply-dml-events
2017-01-04 08:37:08 +02:00
Shlomi Noach
445c67635d
batching DML writes, configurable --dml-batch-size
2017-01-03 14:31:19 +02:00
Shlomi Noach
3e28f462d8
Initial support for batching multiple DMLs when writing to ghost table
2017-01-03 13:44:52 +02:00
Shlomi Noach
c66356bd05
--replication-lag-query is deprecated.
2016-12-26 21:38:37 +02:00
Shlomi Noach
fc831b0548
Reading replication lag via _changelog_ table, also on control replicas
2016-12-26 21:31:35 +02:00
Shlomi Noach
eac990bc95
Merge branch 'master' into explicit-master-user-password
2016-12-15 07:58:12 +01:00
Shlomi Noach
ba2a9d9e55
support for --master-user and --master-password
2016-12-13 16:09:34 +01:00
Shlomi Noach
d12c0265b0
sleeping 500ms after 'START SLAVE'
2016-12-13 12:27:20 +01:00
Shlomi Noach
1d84cb933c
fix: bailing out on no PRIMARY/UNIQUE KEY
2016-12-11 02:19:19 +01:00
Shlomi Noach
35eeb56032
improved log/error messages
2016-12-05 13:41:49 +01:00
rj03hou
8f02ab0fed
check the slave status when find recursive find the master, so support if the dba using reset slave instead of reset slave all.
2016-12-05 19:42:16 +08:00
Shlomi Noach
5904084e14
Merge branch 'master' into myisam-gtid
2016-12-02 15:00:44 +01:00
rj03hou
ffbd35e180
fix TableEngine correlates to the 3rd placeholder in the template string, not the 1st
2016-12-02 11:56:29 +08:00
Shlomi Noach
9b068ec222
Merge branch 'master' into myisam-gtid
2016-12-01 09:43:38 +01:00
rj03hou
a11bec1785
If the original table is MyISAM and the default engine is Innodb, and the gtid mode is on, there will be error when execute 'LOCK TABLES tbl WRITE, tbl_magic WRITE'. If make the magic cut-over table's engine same with the original table, there will not have this problem.
2016-12-01 16:04:04 +08:00
Shlomi Noach
5119ea4d31
added tests to verify no false positives rename-column found
2016-11-29 11:08:35 +01:00
Shlomi Noach
7126b28169
support for --skip-foreign-key-checks
2016-11-21 09:18:40 +01:00
Shlomi Noach
7ab6af8f5f
never throttling inside cut-over critical section
2016-11-17 17:22:13 +01:00
Shlomi Noach
b00cae11fa
retry cut-over
2016-11-17 17:10:17 +01:00
Shlomi Noach
8d987b5aaf
extracted parsing of ChangelogState
2016-11-17 15:56:59 +01:00
Shlomi Noach
ef874b8551
AllEventsUpToLockProcessed uses unique signature
2016-11-17 15:50:54 +01:00
Shlomi Noach
ee447ad560
waitForEventsUpToLock timeout
...
more info on AllEventsUpToLockProcessed, before and after injecting/intercepting
2016-11-17 15:20:44 +01:00
Shlomi Noach
c2d4f624af
simplified code
2016-11-03 12:14:53 +01:00
Shlomi Noach
88ffb75b8c
reading and reporting replication lag before waiting on initial replication event
2016-11-02 12:48:35 +01:00
Shlomi Noach
034683f482
Merge branch 'master' into close-streamer-connection
2016-11-01 12:22:21 +01:00
Shlomi Noach
b32e1e2c16
Merge branch 'master' into throttle-no-changelog-write
2016-11-01 12:20:51 +01:00
Shlomi Noach
bb22431b83
fixed log_slave_updates check logic
2016-10-31 10:25:34 +01:00
Shlomi Noach
7fa5e405d4
avoid writing heartbeat when throttle commanded by user
...
when throttling on user command there really is no need for injecting heartbeat. The user commanded, therefore gh-ost complies and trusts the reasoning for throttling. What this will allow is complete quiet time. This, in turn, will allow such features as relocating via orchestrator/pseudo-gtid at time of throttling
2016-10-27 14:51:38 +02:00
Shlomi Noach
7b63b4a275
proper cleanup of streamer connection
2016-10-27 13:52:37 +02:00
Shlomi Noach
ac6159791d
merged master, resolved conflicts
2016-10-26 09:57:59 +02:00
Shlomi Noach
b696106cb6
fixing bug introduced for charset and timezone tests
2016-10-20 15:05:47 +02:00
Shlomi Noach
bf92eec214
validating table structure on applier and migrator
...
- reading column list on applier
- comparing original table on applier and migrator, expecting exact column list
- or else bailing out
2016-10-20 11:29:30 +02:00
Shlomi Noach
82a76ae879
Merge branch 'master' into enum-sort-test
2016-10-20 10:16:02 +02:00
Akshay Chhajed
559c7895ca
Modified slave update check to test tungsten flag
2016-10-19 23:31:16 +05:30
Shlomi Noach
25166e33c7
solving the enum-as-part-of-pk bug
2016-10-19 15:22:29 +02:00
Shlomi Noach
f9c15127cd
simplified applier read of timezone
2016-10-14 12:56:43 +02:00
Shlomi Noach
9ac468c97e
failing on DATETIME-to-TIMESTAMP conversion within chosen key
2016-10-14 09:33:53 +02:00
Shlomi Noach
0981f3f7d3
fix error message placeholder
2016-10-13 13:10:16 +02:00
Shlomi Noach
84feae052f
inspector notes down TIMESTAMP and DATETIME types; notices in particular the conversion from DATETIME to TIMESTAMP
2016-10-13 13:09:58 +02:00
Shlomi Noach
ac304def4d
applier always uses UTC
2016-10-13 13:08:02 +02:00
Shlomi Noach
184643157b
Merge branch 'master' into tz-a-different-approach
2016-10-12 08:33:19 +02:00
Shlomi Noach
c11c611755
Merge pull request #260 from github/hooks-nil-check
...
safe access to applier/inspector hostnames for hooks
2016-10-12 08:16:50 +02:00
Shlomi Noach
c399119139
Merge pull request #263 from github/implied-key-fix
...
assume-master-host now applies ImpliedKey
2016-10-12 08:16:38 +02:00
Shlomi Noach
4d903d0119
Merge pull request #264 from github/discard-foreign-keys
...
Discard foreign keys
2016-10-12 08:16:27 +02:00
Shlomi Noach
5d92da4a74
Merge branch 'master' into tz-a-different-approach
2016-10-11 17:17:42 +02:00
Shlomi Noach
8bd36dcb01
Merge branch 'master' into implied-key-fix
2016-10-11 17:17:29 +02:00
Shlomi Noach
c1a6773c02
better handling of --assume-master-host
...
separated logic and not even attempting to crawl topology
2016-10-11 16:42:19 +02:00
Shlomi Noach
dbf50afbc7
reading time_zone settings for Inspector and Applier separately.
...
--time-zone overrides both of them, if given
2016-10-11 16:00:26 +02:00
Shlomi Noach
5de8a010df
restoring original row_event code; tampering with applier time_zone based on global timze_zone
2016-10-10 22:33:54 +02:00
Shlomi Noach
15f4ddfd8a
support for --critical-load-interval-millis
...
this optional flag gives --critical-load a second chance. When configured to positive value, meeting with critical-load spawns a timer. When this timer expires a second check for critical-load is made; if still met, gh-ost bails out.
By default the interval is zero, in which case gh-ost bails out immediately on meeting critical load.
2016-10-10 13:21:01 +02:00
Shlomi Noach
a4d566e71c
counting child vs parent sie foreign keys on table,
...
parent-side foreign keys cannot be discarded
2016-10-10 12:28:56 +02:00
Shlomi Noach
6750959e1a
configurable time-zone, native time parsing
2016-10-10 11:39:57 +02:00
Shlomi Noach
0c35a811f7
setting time_zoe='+00:00' on rowcopy
2016-10-08 11:06:27 +02:00
Shlomi Noach
1f65473e69
support for --discard-foreign-keys
...
This flag makes migration silently and happily discard any existing foreign keys on migrated table. This is useful for intentional dropping of foreign keys, as gh-ost does not otherwise have support for foreign key migration.
At some time in the future gh-ost may support foreign key migration, at which time this flag will be removed
2016-10-07 10:20:50 +02:00
Shlomi Noach
ef04fa49f5
assume-master-host now applied ImpliedKey
2016-10-06 12:00:34 +02:00
Shlomi Noach
72f63d3042
safe access to applier/inspector hostnames for hooks
2016-10-04 21:18:44 +02:00
Shlomi Noach
a7627091a7
Merge branch 'master' into named-cut-over
2016-09-13 05:25:16 -07:00
Shlomi Noach
e5e0444cc6
supporting --force-named-cut-over
...
- when given, user _must_ specify table name
and of course table name must match migrated table
2016-09-12 19:17:36 +02:00
Shlomi Noach
0a8be1dd22
excplicitly breaking on NoPrintStatusRule
2016-09-12 17:39:56 +02:00
Shlomi Noach
1c6f828091
refactored server command into server.go
...
- added support for cut-over=<tablename>
- refactored more code into context
2016-09-12 12:38:14 +02:00
Shlomi Noach
16fc19b157
rowcount progress at 100% when row-copy completes
2016-09-12 10:25:55 +02:00
Shlomi Noach
791d963ea0
Character set recognition and manipulation
...
- Identifying textual characters sets; converting into specific type when applying dml events
- Refactored `ColumnsList`: introducing `Column` type
- Refactored `unsigned` handling, as part of `Column`
- `Column` type supports `convertArg()`: converting value of argument according to column data type
- DB URI attempts `utf8mb4,utf8,latin1` charsets in that order (first one to be recognized wins)
- Local tests filter by pattern
- Local tests append table schema on failure
- Local tests do not have postpone flag file
- Added character set local tests: `utf8`, `utf8mb4`, `latin1`
2016-09-07 14:24:11 +02:00
Shlomi Noach
88f2af8111
support for --assume-master-host, master-master/tungsten
2016-09-02 13:09:18 +02:00
Shlomi Noach
96f108d3b4
Merge pull request #221 from twotwotwo/override-applier-host
...
Add -override-applier-host for use with -allow-master-master
2016-09-02 11:32:04 +02:00
Shlomi Noach
a58e80387e
More environment variables in hooks
2016-09-02 10:48:29 +02:00
Shlomi Noach
75d225353f
Merge pull request #220 from Wattpad/exit-on-hook-replication-stop-failure
...
Fail operation if onStopReplication hook fails
2016-09-02 09:39:43 +02:00
randall
82110fcfcf
Add -override-applier-host for use with -allow-master-master
...
for configurations where writes are meant to go to one master, but gh-ost can't automatically determine which
2016-09-01 20:29:26 -07:00
Paulo Bittencourt
e3662f2398
Fail operation if onStopReplication hook fails
2016-09-01 15:58:20 -04:00
Shlomi Noach
c562df42cd
status: State and ETA decoupling
2016-09-01 10:51:40 +02:00
Shlomi Noach
904215e286
Merge pull request #204 from github/reduce-minimum-max-lag
...
Reduce minimum maxLagMillisecondsThrottleThreshold to 100ms
2016-08-31 09:29:16 +02:00
Shlomi Noach
aef56c55f7
indicating 100% when rowcopy is complete
2016-08-30 17:02:29 +02:00
Shlomi Noach
b2c71931c6
refactored all throttling code into throttler.so
2016-08-30 12:25:45 +02:00
Shlomi Noach
23357d0643
WIP: decoupling general throttling from throttle logic
2016-08-30 11:32:17 +02:00
Shlomi Noach
75b2542f26
Merge branch 'master' into reduce-minimum-max-lag
2016-08-30 09:47:33 +02:00
Shlomi Noach
1c3699cc64
fixed import path
2016-08-30 09:45:02 +02:00
Shlomi Noach
2afb86b9e4
support for millisecond throttling
...
- `--max-lag-millis` is at least `100ms`
- `--heartbeat-interval-millis` introduced; defaults `500ms`, can range `100ms` - `1s`
- Control replicas lag calculated asynchronously to throttle test
- aggressive when `max-lag-millis < 1000` and when `replication-lag-query` is given
2016-08-30 09:41:59 +02:00
Shlomi Noach
176aabead0
Merge pull request #190 from github/hooks
...
WIP: Hooks
2016-08-30 09:01:19 +02:00
Shlomi Noach
c0936ed728
clarified privileges requirements
2016-08-29 12:39:31 +02:00
Shlomi Noach
6dfa4873c2
removed excessive argument
2016-08-29 10:44:43 +02:00
Shlomi Noach
6e5db089c8
supporting onRowCountComplete hook
2016-08-29 09:58:31 +02:00
Shlomi Noach
c70f405d06
Merge branch 'master' into hooks
2016-08-26 08:39:02 +02:00
Shlomi Noach
4f9c1448d1
Merge pull request #200 from github/support-5-5
...
fixed 5.5 binlog_row_image default value
2016-08-26 08:29:33 +02:00
Shlomi Noach
2b595b15f2
Merge pull request #196 from github/concurrent-rowcount
...
concurrent row-count
2016-08-26 08:29:01 +02:00
Shlomi Noach
63a0e5cf2c
fixed 5.5 binlog_row_image default value
2016-08-25 16:49:25 +02:00
Shlomi Noach
ef386add5c
hook output and err are written to gh-ost's stderr
2016-08-25 16:08:49 +02:00
Shlomi Noach
2c9d71570f
logging intented hook invocation
2016-08-25 14:42:49 +02:00
Shlomi Noach
ecd33969bf
added onRowCountComplete(); supporting elapsedSeconds in on-status
2016-08-25 13:55:22 +02:00
Shlomi Noach
b064174ab4
added elapsedSeconds to status hook
2016-08-25 13:54:21 +02:00
Shlomi Noach
cb1a7e2805
merged master
2016-08-25 12:32:03 +02:00
Shlomi Noach
c7d88499af
Merge branch 'master' into row-copy-complete
2016-08-25 10:15:32 +02:00
Shlomi Noach
c7edd1ef84
Merge branch 'master' into concurrent-rowcount
2016-08-25 09:44:12 +02:00
Shlomi Noach
1773f338c2
keeping track of delta rows
...
on concurrent count(*) this means we re-apply delta onto new estimate
2016-08-24 12:16:34 +02:00
Shlomi Noach
553f4c8d13
concurrent row-count
2016-08-24 11:39:44 +02:00
Shlomi Noach
4c972184a8
Merge branch 'master' into sql-mode-strict
2016-08-24 09:34:00 +02:00
Shlomi Noach
56fd82a824
Merge pull request #174 from Wattpad/test-on-replica-manual-replication-control
...
outstanding. Thank you!
2016-08-24 09:12:21 +02:00
Paulo Bittencourt
6b21ade6d0
Check for --test-on-replica-skip-replica-stop in cutOver method
2016-08-23 18:34:10 -04:00
Shlomi Noach
56d09c4105
avoiding writing rows when rowcopy complete
2016-08-23 14:26:47 +02:00
Shlomi Noach
8b76d0e75b
DML write sets sql_mode to STRICT ALL TABLES
2016-08-23 11:58:52 +02:00
Shlomi Noach
a1e191078a
rename 'about'->'before'
2016-08-23 11:40:32 +02:00
Shlomi Noach
1c2a77ef95
hook names; added on-stop-replication hook
2016-08-23 11:35:48 +02:00
Shlomi Noach
1021a83ac0
Merge pull request #189 from dveeden/feedback_on_wait
...
thank you
2016-08-23 09:50:50 +02:00
Daniël van Eeden
d8cfd49e2c
Message about waiting should be INFO not DEBUG
2016-08-23 09:41:07 +02:00
Shlomi Noach
972728cf40
added onStatus hook
2016-08-22 16:24:41 +02:00
Shlomi Noach
b63cc3e75e
fix INSERT DML handling on renamed column
2016-08-22 16:00:15 +02:00
Shlomi Noach
9cf4819a98
Merge branch 'master' into fix-rename
...
Wish to incorporate important time_zone fix
2016-08-22 11:54:52 +02:00