octoleo/gh-ost
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
725 Commits 51 Branches 51 Tags 24 MiB
109b82bb3a
Commit Graph

312 Commits

Author SHA1 Message Date
Shlomi Noach
034683f482 Merge branch 'master' into close-streamer-connection 2016-11-01 12:22:21 +01:00
Shlomi Noach
b32e1e2c16 Merge branch 'master' into throttle-no-changelog-write 2016-11-01 12:20:51 +01:00
Shlomi Noach
bb22431b83 fixed log_slave_updates check logic 2016-10-31 10:25:34 +01:00
Shlomi Noach
10850e4346 Merge branch 'master' into concurrent-rowcount-defaults-true 2016-10-28 10:14:22 +02:00
Shlomi Noach
349c9af08d Merge branch 'master' into close-streamer-connection 2016-10-28 09:30:16 +02:00
Shlomi Noach
f2bfaedcd6 Merge branch 'master' into throttle-no-changelog-write 2016-10-28 09:30:00 +02:00
Shlomi Noach
5aad45e3bc formatted code via gofmt 2016-10-27 20:38:45 +02:00
Shlomi Noach
7fa5e405d4 avoid writing heartbeat when throttle commanded by user 
when throttling on user command there really is no need for injecting heartbeat. The user commanded, therefore gh-ost complies and trusts the reasoning for throttling. What this will allow is complete quiet time. This, in turn, will allow such features as relocating via orchestrator/pseudo-gtid at time of throttling
 2016-10-27 14:51:38 +02:00
Shlomi Noach
7b63b4a275 proper cleanup of streamer connection 2016-10-27 13:52:37 +02:00
Shlomi Noach
7fe7b032e9 concurrent-rowcount defaults 'true' 2016-10-26 10:40:12 +02:00
Shlomi Noach
ac6159791d merged master, resolved conflicts 2016-10-26 09:57:59 +02:00
Shlomi Noach
b696106cb6 fixing bug introduced for charset and timezone tests 2016-10-20 15:05:47 +02:00
Shlomi Noach
bf92eec214 validating table structure on applier and migrator 
- reading column list on applier
- comparing original table on applier and migrator, expecting exact column list
- or else bailing out
 2016-10-20 11:29:30 +02:00
Shlomi Noach
82a76ae879 Merge branch 'master' into enum-sort-test 2016-10-20 10:16:02 +02:00
Akshay Chhajed
559c7895ca Modified slave update check to test tungsten flag 2016-10-19 23:31:16 +05:30
Shlomi Noach
25166e33c7 solving the enum-as-part-of-pk bug 2016-10-19 15:22:29 +02:00
Shlomi Noach
f9c15127cd simplified applier read of timezone 2016-10-14 12:56:43 +02:00
Shlomi Noach
dcd2f36c88 removed --time-zone flag 2016-10-14 12:53:23 +02:00
Shlomi Noach
9ac468c97e failing on DATETIME-to-TIMESTAMP conversion within chosen key 2016-10-14 09:33:53 +02:00
Shlomi Noach
11ccf09412 grammar 2016-10-14 09:07:30 +02:00
Shlomi Noach
e6d317b579 fixed inferred usage for --critical-load 2016-10-14 09:05:54 +02:00
Shlomi Noach
661b7ff0c1 added test to sql/types 2016-10-13 13:13:00 +02:00
Shlomi Noach
882959ba83 column is type-aware; is timezone-conversion aware 
also refactored some functions
 2016-10-13 13:12:33 +02:00
Shlomi Noach
a637ed1b50 adapted builder tests 2016-10-13 13:11:35 +02:00
Shlomi Noach
6fbf337051 builder is smarter and can use convert_tz where it notices DATETIME->TIMESTAMP conversion 2016-10-13 13:11:18 +02:00
Shlomi Noach
0981f3f7d3 fix error message placeholder 2016-10-13 13:10:16 +02:00
Shlomi Noach
84feae052f inspector notes down TIMESTAMP and DATETIME types; notices in particular the conversion from DATETIME to TIMESTAMP 2016-10-13 13:09:58 +02:00
Shlomi Noach
d80694bdac no need for Inspector timezone; it is irrelevant 2016-10-13 13:08:41 +02:00
Shlomi Noach
f988309800 no need for Inspector timezone; it is irrelevant 2016-10-13 13:08:23 +02:00
Shlomi Noach
ac304def4d applier always uses UTC 2016-10-13 13:08:02 +02:00
Shlomi Noach
184643157b Merge branch 'master' into tz-a-different-approach 2016-10-12 08:33:19 +02:00
Shlomi Noach
c11c611755 Merge pull request #260 from github/hooks-nil-check 
safe access to applier/inspector hostnames for hooks
 2016-10-12 08:16:50 +02:00
Shlomi Noach
c399119139 Merge pull request #263 from github/implied-key-fix 
assume-master-host now applies ImpliedKey
 2016-10-12 08:16:38 +02:00
Shlomi Noach
4d903d0119 Merge pull request #264 from github/discard-foreign-keys 
Discard foreign keys
 2016-10-12 08:16:27 +02:00
Shlomi Noach
5d92da4a74 Merge branch 'master' into tz-a-different-approach 2016-10-11 17:17:42 +02:00
Shlomi Noach
8bd36dcb01 Merge branch 'master' into implied-key-fix 2016-10-11 17:17:29 +02:00
Shlomi Noach
c1a6773c02 better handling of --assume-master-host 
separated logic and not even attempting to crawl topology
 2016-10-11 16:42:19 +02:00
Shlomi Noach
dbf50afbc7 reading time_zone settings for Inspector and Applier separately. 
--time-zone overrides both of them, if given
 2016-10-11 16:00:26 +02:00
Shlomi Noach
5de8a010df restoring original row_event code; tampering with applier time_zone based on global timze_zone 2016-10-10 22:33:54 +02:00
Shlomi Noach
15f4ddfd8a support for --critical-load-interval-millis 
this optional flag gives --critical-load a second chance. When configured to positive value, meeting with critical-load spawns a timer. When this timer expires a second check for critical-load is made; if still met, gh-ost bails out.
By default the interval is zero, in which case gh-ost bails out immediately on meeting critical load.
 2016-10-10 13:21:01 +02:00
Shlomi Noach
a4d566e71c counting child vs parent sie foreign keys on table, 
parent-side foreign keys cannot be discarded
 2016-10-10 12:28:56 +02:00
Shlomi Noach
6750959e1a configurable time-zone, native time parsing 2016-10-10 11:39:57 +02:00
Shlomi Noach
0c35a811f7 setting time_zoe='+00:00' on rowcopy 2016-10-08 11:06:27 +02:00
Shlomi Noach
1f65473e69 support for --discard-foreign-keys 
This flag makes migration silently and happily discard any existing foreign keys on migrated table. This is useful for intentional dropping of foreign keys, as gh-ost does not otherwise have support for foreign key migration.
At some time in the future gh-ost may support foreign key migration, at which time this flag will be removed
 2016-10-07 10:20:50 +02:00
Shlomi Noach
ef04fa49f5 assume-master-host now applied ImpliedKey 2016-10-06 12:00:34 +02:00
Shlomi Noach
72f63d3042 safe access to applier/inspector hostnames for hooks 2016-10-04 21:18:44 +02:00
Shlomi Noach
51f0c0d957 fix arg conversion in BuildDMLUpdateQuery 2016-09-27 13:08:31 +02:00
Paulo Bittencourt
2f80c9d424 Used the source column definition to fetch and convert insert values 
When processing binlog insert statements, we want to use `sharedColumns`
to decide which values to fetch and convert from the insert DML event.

We only want to  `mappedShareColumns` to define the column names in the
`replace into ...` statement.
 2016-09-14 11:28:30 -04:00
Shlomi Noach
a7627091a7 Merge branch 'master' into named-cut-over 2016-09-13 05:25:16 -07:00
Shlomi Noach
e5e0444cc6 supporting --force-named-cut-over 
- when given, user _must_ specify table name
  and of course table name must match migrated table
 2016-09-12 19:17:36 +02:00
Shlomi Noach
0a8be1dd22 excplicitly breaking on NoPrintStatusRule 2016-09-12 17:39:56 +02:00
Shlomi Noach
1c6f828091 refactored server command into server.go 
- added support for cut-over=<tablename>
- refactored more code into context
 2016-09-12 12:38:14 +02:00
Shlomi Noach
16fc19b157 rowcount progress at 100% when row-copy completes 2016-09-12 10:25:55 +02:00
Shlomi Noach
f4693274fb Extracted and generalized encoding logic to encoding.go 2016-09-08 09:37:22 +02:00
Shlomi Noach
791d963ea0 Character set recognition and manipulation 
- Identifying textual characters sets; converting into specific type when applying dml events
- Refactored `ColumnsList`: introducing `Column` type
- Refactored `unsigned` handling, as part of `Column`
- `Column` type supports `convertArg()`: converting value of argument according to column data type
- DB URI attempts `utf8mb4,utf8,latin1` charsets in that order (first one to be recognized wins)
- Local tests filter by pattern
- Local tests append table schema on failure
- Local tests do not have postpone flag file
- Added character set local tests: `utf8`, `utf8mb4`, `latin1`
 2016-09-07 14:24:11 +02:00
Shlomi Noach
88f2af8111 support for --assume-master-host, master-master/tungsten 2016-09-02 13:09:18 +02:00
Shlomi Noach
96f108d3b4 Merge pull request #221 from twotwotwo/override-applier-host 
Add -override-applier-host for use with -allow-master-master
 2016-09-02 11:32:04 +02:00
Shlomi Noach
a58e80387e More environment variables in hooks 2016-09-02 10:48:29 +02:00
Shlomi Noach
75d225353f Merge pull request #220 from Wattpad/exit-on-hook-replication-stop-failure 
Fail operation if onStopReplication hook fails
 2016-09-02 09:39:43 +02:00
randall
82110fcfcf Add -override-applier-host for use with -allow-master-master 
for configurations where writes are meant to go to one master, but gh-ost can't automatically determine which
 2016-09-01 20:29:26 -07:00
Paulo Bittencourt
e3662f2398 Fail operation if onStopReplication hook fails 2016-09-01 15:58:20 -04:00
Shlomi Noach
c562df42cd status: State and ETA decoupling 2016-09-01 10:51:40 +02:00
Shlomi Noach
904215e286 Merge pull request #204 from github/reduce-minimum-max-lag 
Reduce minimum maxLagMillisecondsThrottleThreshold to 100ms
 2016-08-31 09:29:16 +02:00
Shlomi Noach
aef56c55f7 indicating 100% when rowcopy is complete 2016-08-30 17:02:29 +02:00
Shlomi Noach
b2c71931c6 refactored all throttling code into throttler.so 2016-08-30 12:25:45 +02:00
Shlomi Noach
23357d0643 WIP: decoupling general throttling from throttle logic 2016-08-30 11:32:17 +02:00
Shlomi Noach
75b2542f26 Merge branch 'master' into reduce-minimum-max-lag 2016-08-30 09:47:33 +02:00
Shlomi Noach
1c3699cc64 fixed import path 2016-08-30 09:45:02 +02:00
Shlomi Noach
2afb86b9e4 support for millisecond throttling 
- `--max-lag-millis` is at least `100ms`
- `--heartbeat-interval-millis` introduced; defaults `500ms`, can range `100ms` - `1s`
- Control replicas lag calculated asynchronously to throttle test
  - aggressive when `max-lag-millis < 1000` and when `replication-lag-query` is given
 2016-08-30 09:41:59 +02:00
Shlomi Noach
176aabead0 Merge pull request #190 from github/hooks 
WIP: Hooks
 2016-08-30 09:01:19 +02:00
Shlomi Noach
c0936ed728 clarified privileges requirements 2016-08-29 12:39:31 +02:00
Shlomi Noach
6dfa4873c2 removed excessive argument 2016-08-29 10:44:43 +02:00
Shlomi Noach
6e5db089c8 supporting onRowCountComplete hook 2016-08-29 09:58:31 +02:00
Jonah Berquist
10b222bc7b Reduce minimum maxLagMillisecondsThrottleThreshold to 100ms 2016-08-26 16:44:40 -07:00
Shlomi Noach
c70f405d06 Merge branch 'master' into hooks 2016-08-26 08:39:02 +02:00
Shlomi Noach
4f9c1448d1 Merge pull request #200 from github/support-5-5 
fixed 5.5 binlog_row_image default value
 2016-08-26 08:29:33 +02:00
Shlomi Noach
2b595b15f2 Merge pull request #196 from github/concurrent-rowcount 
concurrent row-count
 2016-08-26 08:29:01 +02:00
Shlomi Noach
63a0e5cf2c fixed 5.5 binlog_row_image default value 2016-08-25 16:49:25 +02:00
Shlomi Noach
ef386add5c hook output and err are written to gh-ost's stderr 2016-08-25 16:08:49 +02:00
Shlomi Noach
2c9d71570f logging intented hook invocation 2016-08-25 14:42:49 +02:00
Shlomi Noach
cbdec6bd39 supporting --hooks-path and --hooks-hint 2016-08-25 13:58:36 +02:00
Shlomi Noach
ecd33969bf added onRowCountComplete(); supporting elapsedSeconds in on-status 2016-08-25 13:55:22 +02:00
Shlomi Noach
fad6743150 added hooks hint message 2016-08-25 13:54:42 +02:00
Shlomi Noach
b064174ab4 added elapsedSeconds to status hook 2016-08-25 13:54:21 +02:00
Shlomi Noach
cb1a7e2805 merged master 2016-08-25 12:32:03 +02:00
Shlomi Noach
c7d88499af Merge branch 'master' into row-copy-complete 2016-08-25 10:15:32 +02:00
Shlomi Noach
c7edd1ef84 Merge branch 'master' into concurrent-rowcount 2016-08-25 09:44:12 +02:00
Shlomi Noach
1773f338c2 keeping track of delta rows 
on concurrent count(*) this means we re-apply delta onto new estimate
 2016-08-24 12:16:34 +02:00
Shlomi Noach
553f4c8d13 concurrent row-count 2016-08-24 11:39:44 +02:00
Shlomi Noach
4c972184a8 Merge branch 'master' into sql-mode-strict 2016-08-24 09:34:00 +02:00
Shlomi Noach
56fd82a824 Merge pull request #174 from Wattpad/test-on-replica-manual-replication-control 
outstanding. Thank you!
 2016-08-24 09:12:21 +02:00
Paulo Bittencourt
6b21ade6d0 Check for --test-on-replica-skip-replica-stop in cutOver method 2016-08-23 18:34:10 -04:00
Shlomi Noach
56d09c4105 avoiding writing rows when rowcopy complete 2016-08-23 14:26:47 +02:00
Shlomi Noach
8b76d0e75b DML write sets sql_mode to STRICT ALL TABLES 2016-08-23 11:58:52 +02:00
Shlomi Noach
a1e191078a rename 'about'->'before' 2016-08-23 11:40:32 +02:00
Shlomi Noach
1c2a77ef95 hook names; added on-stop-replication hook 2016-08-23 11:35:48 +02:00
Shlomi Noach
1021a83ac0 Merge pull request #189 from dveeden/feedback_on_wait 
thank you
 2016-08-23 09:50:50 +02:00
Daniël van Eeden
d8cfd49e2c Message about waiting should be INFO not DEBUG 2016-08-23 09:41:07 +02:00
Shlomi Noach
972728cf40 added onStatus hook 2016-08-22 16:24:41 +02:00
Shlomi Noach
b63cc3e75e fix INSERT DML handling on renamed column 2016-08-22 16:00:15 +02:00