Commit Graph

345 Commits

Author SHA1 Message Date
Shlomi Noach
dcd2f36c88 removed --time-zone flag 2016-10-14 12:53:23 +02:00
Shlomi Noach
9ac468c97e failing on DATETIME-to-TIMESTAMP conversion within chosen key 2016-10-14 09:33:53 +02:00
Shlomi Noach
11ccf09412 grammar 2016-10-14 09:07:30 +02:00
Shlomi Noach
e6d317b579 fixed inferred usage for --critical-load 2016-10-14 09:05:54 +02:00
Shlomi Noach
661b7ff0c1 added test to sql/types 2016-10-13 13:13:00 +02:00
Shlomi Noach
882959ba83 column is type-aware; is timezone-conversion aware
also refactored some functions
2016-10-13 13:12:33 +02:00
Shlomi Noach
a637ed1b50 adapted builder tests 2016-10-13 13:11:35 +02:00
Shlomi Noach
6fbf337051 builder is smarter and can use convert_tz where it notices DATETIME->TIMESTAMP conversion 2016-10-13 13:11:18 +02:00
Shlomi Noach
0981f3f7d3 fix error message placeholder 2016-10-13 13:10:16 +02:00
Shlomi Noach
84feae052f inspector notes down TIMESTAMP and DATETIME types; notices in particular the conversion from DATETIME to TIMESTAMP 2016-10-13 13:09:58 +02:00
Shlomi Noach
d80694bdac no need for Inspector timezone; it is irrelevant 2016-10-13 13:08:41 +02:00
Shlomi Noach
f988309800 no need for Inspector timezone; it is irrelevant 2016-10-13 13:08:23 +02:00
Shlomi Noach
ac304def4d applier always uses UTC 2016-10-13 13:08:02 +02:00
Shlomi Noach
184643157b Merge branch 'master' into tz-a-different-approach 2016-10-12 08:33:19 +02:00
Shlomi Noach
c11c611755 Merge pull request #260 from github/hooks-nil-check
safe access to applier/inspector hostnames for hooks
2016-10-12 08:16:50 +02:00
Shlomi Noach
c399119139 Merge pull request #263 from github/implied-key-fix
assume-master-host now applies ImpliedKey
2016-10-12 08:16:38 +02:00
Shlomi Noach
4d903d0119 Merge pull request #264 from github/discard-foreign-keys
Discard foreign keys
2016-10-12 08:16:27 +02:00
Shlomi Noach
5d92da4a74 Merge branch 'master' into tz-a-different-approach 2016-10-11 17:17:42 +02:00
Shlomi Noach
8bd36dcb01 Merge branch 'master' into implied-key-fix 2016-10-11 17:17:29 +02:00
Shlomi Noach
c1a6773c02 better handling of --assume-master-host
separated logic and not even attempting to crawl topology
2016-10-11 16:42:19 +02:00
Shlomi Noach
dbf50afbc7 reading time_zone settings for Inspector and Applier separately.
--time-zone overrides both of them, if given
2016-10-11 16:00:26 +02:00
Shlomi Noach
5de8a010df restoring original row_event code; tampering with applier time_zone based on global timze_zone 2016-10-10 22:33:54 +02:00
Shlomi Noach
15f4ddfd8a support for --critical-load-interval-millis
this optional flag gives --critical-load a second chance. When configured to positive value, meeting with critical-load spawns a timer. When this timer expires a second check for critical-load is made; if still met, gh-ost bails out.
By default the interval is zero, in which case gh-ost bails out immediately on meeting critical load.
2016-10-10 13:21:01 +02:00
Shlomi Noach
a4d566e71c counting child vs parent sie foreign keys on table,
parent-side foreign keys cannot be discarded
2016-10-10 12:28:56 +02:00
Shlomi Noach
6750959e1a configurable time-zone, native time parsing 2016-10-10 11:39:57 +02:00
Shlomi Noach
0c35a811f7 setting time_zoe='+00:00' on rowcopy 2016-10-08 11:06:27 +02:00
Shlomi Noach
1f65473e69 support for --discard-foreign-keys
This flag makes migration silently and happily discard any existing foreign keys on migrated table. This is useful for intentional dropping of foreign keys, as gh-ost does not otherwise have support for foreign key migration.
At some time in the future gh-ost may support foreign key migration, at which time this flag will be removed
2016-10-07 10:20:50 +02:00
Shlomi Noach
ef04fa49f5 assume-master-host now applied ImpliedKey 2016-10-06 12:00:34 +02:00
Shlomi Noach
72f63d3042 safe access to applier/inspector hostnames for hooks 2016-10-04 21:18:44 +02:00
Shlomi Noach
51f0c0d957 fix arg conversion in BuildDMLUpdateQuery 2016-09-27 13:08:31 +02:00
Paulo Bittencourt
2f80c9d424 Used the source column definition to fetch and convert insert values
When processing binlog insert statements, we want to use `sharedColumns`
to decide which values to fetch and convert from the insert DML event.

We only want to  `mappedShareColumns` to define the column names in the
`replace into ...` statement.
2016-09-14 11:28:30 -04:00
Shlomi Noach
a7627091a7 Merge branch 'master' into named-cut-over 2016-09-13 05:25:16 -07:00
Shlomi Noach
e5e0444cc6 supporting --force-named-cut-over
- when given, user _must_ specify table name
  and of course table name must match migrated table
2016-09-12 19:17:36 +02:00
Shlomi Noach
0a8be1dd22 excplicitly breaking on NoPrintStatusRule 2016-09-12 17:39:56 +02:00
Shlomi Noach
1c6f828091 refactored server command into server.go
- added support for cut-over=<tablename>
- refactored more code into context
2016-09-12 12:38:14 +02:00
Shlomi Noach
16fc19b157 rowcount progress at 100% when row-copy completes 2016-09-12 10:25:55 +02:00
Shlomi Noach
f4693274fb Extracted and generalized encoding logic to encoding.go 2016-09-08 09:37:22 +02:00
Shlomi Noach
791d963ea0 Character set recognition and manipulation
- Identifying textual characters sets; converting into specific type when applying dml events
- Refactored `ColumnsList`: introducing `Column` type
- Refactored `unsigned` handling, as part of `Column`
- `Column` type supports `convertArg()`: converting value of argument according to column data type
- DB URI attempts `utf8mb4,utf8,latin1` charsets in that order (first one to be recognized wins)
- Local tests filter by pattern
- Local tests append table schema on failure
- Local tests do not have postpone flag file
- Added character set local tests: `utf8`, `utf8mb4`, `latin1`
2016-09-07 14:24:11 +02:00
Shlomi Noach
88f2af8111 support for --assume-master-host, master-master/tungsten 2016-09-02 13:09:18 +02:00
Shlomi Noach
96f108d3b4 Merge pull request #221 from twotwotwo/override-applier-host
Add -override-applier-host for use with -allow-master-master
2016-09-02 11:32:04 +02:00
Shlomi Noach
a58e80387e More environment variables in hooks 2016-09-02 10:48:29 +02:00
Shlomi Noach
75d225353f Merge pull request #220 from Wattpad/exit-on-hook-replication-stop-failure
Fail operation if onStopReplication hook fails
2016-09-02 09:39:43 +02:00
randall
82110fcfcf Add -override-applier-host for use with -allow-master-master
for configurations where writes are meant to go to one master, but gh-ost can't automatically determine which
2016-09-01 20:29:26 -07:00
Paulo Bittencourt
e3662f2398 Fail operation if onStopReplication hook fails 2016-09-01 15:58:20 -04:00
Shlomi Noach
c562df42cd status: State and ETA decoupling 2016-09-01 10:51:40 +02:00
Shlomi Noach
904215e286 Merge pull request #204 from github/reduce-minimum-max-lag
Reduce minimum maxLagMillisecondsThrottleThreshold to 100ms
2016-08-31 09:29:16 +02:00
Shlomi Noach
aef56c55f7 indicating 100% when rowcopy is complete 2016-08-30 17:02:29 +02:00
Shlomi Noach
b2c71931c6 refactored all throttling code into throttler.so 2016-08-30 12:25:45 +02:00
Shlomi Noach
23357d0643 WIP: decoupling general throttling from throttle logic 2016-08-30 11:32:17 +02:00
Shlomi Noach
75b2542f26 Merge branch 'master' into reduce-minimum-max-lag 2016-08-30 09:47:33 +02:00
Shlomi Noach
1c3699cc64 fixed import path 2016-08-30 09:45:02 +02:00
Shlomi Noach
2afb86b9e4 support for millisecond throttling
- `--max-lag-millis` is at least `100ms`
- `--heartbeat-interval-millis` introduced; defaults `500ms`, can range `100ms` - `1s`
- Control replicas lag calculated asynchronously to throttle test
  - aggressive when `max-lag-millis < 1000` and when `replication-lag-query` is given
2016-08-30 09:41:59 +02:00
Shlomi Noach
176aabead0 Merge pull request #190 from github/hooks
WIP: Hooks
2016-08-30 09:01:19 +02:00
Shlomi Noach
c0936ed728 clarified privileges requirements 2016-08-29 12:39:31 +02:00
Shlomi Noach
6dfa4873c2 removed excessive argument 2016-08-29 10:44:43 +02:00
Shlomi Noach
6e5db089c8 supporting onRowCountComplete hook 2016-08-29 09:58:31 +02:00
Jonah Berquist
10b222bc7b Reduce minimum maxLagMillisecondsThrottleThreshold to 100ms 2016-08-26 16:44:40 -07:00
Shlomi Noach
c70f405d06 Merge branch 'master' into hooks 2016-08-26 08:39:02 +02:00
Shlomi Noach
4f9c1448d1 Merge pull request #200 from github/support-5-5
fixed 5.5 binlog_row_image default value
2016-08-26 08:29:33 +02:00
Shlomi Noach
2b595b15f2 Merge pull request #196 from github/concurrent-rowcount
concurrent row-count
2016-08-26 08:29:01 +02:00
Shlomi Noach
63a0e5cf2c fixed 5.5 binlog_row_image default value 2016-08-25 16:49:25 +02:00
Shlomi Noach
ef386add5c hook output and err are written to gh-ost's stderr 2016-08-25 16:08:49 +02:00
Shlomi Noach
2c9d71570f logging intented hook invocation 2016-08-25 14:42:49 +02:00
Shlomi Noach
cbdec6bd39 supporting --hooks-path and --hooks-hint 2016-08-25 13:58:36 +02:00
Shlomi Noach
ecd33969bf added onRowCountComplete(); supporting elapsedSeconds in on-status 2016-08-25 13:55:22 +02:00
Shlomi Noach
fad6743150 added hooks hint message 2016-08-25 13:54:42 +02:00
Shlomi Noach
b064174ab4 added elapsedSeconds to status hook 2016-08-25 13:54:21 +02:00
Shlomi Noach
cb1a7e2805 merged master 2016-08-25 12:32:03 +02:00
Shlomi Noach
c7d88499af Merge branch 'master' into row-copy-complete 2016-08-25 10:15:32 +02:00
Shlomi Noach
c7edd1ef84 Merge branch 'master' into concurrent-rowcount 2016-08-25 09:44:12 +02:00
Shlomi Noach
1773f338c2 keeping track of delta rows
on concurrent count(*) this means we re-apply delta onto new estimate
2016-08-24 12:16:34 +02:00
Shlomi Noach
553f4c8d13 concurrent row-count 2016-08-24 11:39:44 +02:00
Shlomi Noach
4c972184a8 Merge branch 'master' into sql-mode-strict 2016-08-24 09:34:00 +02:00
Shlomi Noach
56fd82a824 Merge pull request #174 from Wattpad/test-on-replica-manual-replication-control
outstanding. Thank you!
2016-08-24 09:12:21 +02:00
Paulo Bittencourt
6b21ade6d0 Check for --test-on-replica-skip-replica-stop in cutOver method 2016-08-23 18:34:10 -04:00
Shlomi Noach
56d09c4105 avoiding writing rows when rowcopy complete 2016-08-23 14:26:47 +02:00
Shlomi Noach
8b76d0e75b DML write sets sql_mode to STRICT ALL TABLES 2016-08-23 11:58:52 +02:00
Shlomi Noach
a1e191078a rename 'about'->'before' 2016-08-23 11:40:32 +02:00
Shlomi Noach
1c2a77ef95 hook names; added on-stop-replication hook 2016-08-23 11:35:48 +02:00
Shlomi Noach
1021a83ac0 Merge pull request #189 from dveeden/feedback_on_wait
thank you
2016-08-23 09:50:50 +02:00
Daniël van Eeden
d8cfd49e2c Message about waiting should be INFO not DEBUG 2016-08-23 09:41:07 +02:00
Shlomi Noach
972728cf40 added onStatus hook 2016-08-22 16:24:41 +02:00
Shlomi Noach
b63cc3e75e fix INSERT DML handling on renamed column 2016-08-22 16:00:15 +02:00
Shlomi Noach
9cf4819a98 Merge branch 'master' into fix-rename
Wish to incorporate important time_zone fix
2016-08-22 11:54:52 +02:00
Shlomi Noach
1376f0af23 fixed UPDATE dml on renamed column 2016-08-22 08:49:27 +02:00
Shlomi Noach
fc954562a3 Merge pull request #175 from github/apply-dml-tz
setting time_zone on DML apply
2016-08-22 08:01:50 +02:00
Shlomi Noach
6acbe7e3ae detecting and executing hooks 2016-08-20 08:24:20 +02:00
Paulo Bittencourt
2e43718ef3 Add --test-on-replica-skip-replica-stop flag 2016-08-19 17:34:08 -04:00
Shlomi Noach
cdf393a30e initial support for hooks 2016-08-19 14:52:49 +02:00
Shlomi Noach
d8e30fcd85 fixed sup printing heuristic 2016-08-19 09:41:25 +02:00
Shlomi Noach
9752179723 interactive command: sup 2016-08-19 09:16:17 +02:00
Shlomi Noach
6d80340e4f setting time_zone on DML apply 2016-08-19 09:06:00 +02:00
Shlomi Noach
e6a02d81e0 Merge pull request #170 from github/nice-ratio-doc-clarification
clarifying meaning of sleep-ratio
2016-08-19 08:27:22 +02:00
Shlomi Noach
1e88dbd04a Merge pull request #171 from github/empty-table-progress
progress is 100% when 0/0 rows copied
2016-08-19 08:27:04 +02:00
Shlomi Noach
4a7df07f64 Merge pull request #173 from github/accept-mysql-prompt-config
relaxed config scanner mode
2016-08-19 08:26:13 +02:00
Shlomi Noach
41d43933c8 Merge pull request #172 from github/transactional-apply-dml
DML write wrapped in transaction
2016-08-19 08:25:52 +02:00
Paulo Bittencourt
a62f9e0754 Add --test-on-replica-manual-replication-control flag
This will wait indefinitely for the replication status to change.
This allows us to run test schema changes in RDS without needing
custom RDS commands in gh-ost.
2016-08-18 11:53:25 -04:00
Shlomi Noach
00369d7e5d relaxed config scanner mode
- does not fail on MySQL 'prompt' config
2016-08-18 13:58:38 +02:00
Shlomi Noach
88eb2d6ee1 Merge pull request #130 from dveeden/foreign_key_info
Give more info about foreign keys
2016-08-18 13:40:31 +02:00
Shlomi Noach
75e0d12302 simplified error logic; fixed incorrect RowsEstimate handling on error 2016-08-18 13:38:23 +02:00
Shlomi Noach
74593ec010 DML write wrapped in transaction
- solving the golang problem: 'sql: converting Exec argument #2's type: uint64 values with high bit set are not supported'
2016-08-18 13:31:53 +02:00
Shlomi Noach
7e9f578e12 progress is 100% when 0/0 rows copied 2016-08-18 13:20:09 +02:00
Shlomi Noach
5dbd2e1c85 clarifying meaning of sleep-ratio 2016-08-18 13:13:51 +02:00
Shlomi Noach
d9ae2f3942 Merge pull request #156 from github/avoid-restarting-replication
accepting `--assume-rbr`, remove `SUPER` requirement
2016-08-18 10:11:35 +02:00
Daniël van Eeden
d3422bd19a Give more info about foreign keys
This helps when the table itself doesn't have foreign keys defined but if there are other tables with foreign keys pointing to the table on which gh-ost runs.

This gives INFO messages for each FK. Note that it now informs the user about `payment` being involved.
```
$ ./gh-ost -database sakila -table rental -alter 'ADD COLUMN ghost_test_001 tinyint DEFAULT NULL' -port 19590 -user msandbox -password msandbox -verbose
2016-08-03 10:18:45 INFO starting gh-ost 1.0.8
2016-08-03 10:18:45 INFO Migrating `sakila`.`rental`
2016-08-03 10:18:45 INFO connection validated on 127.0.0.1:19590
2016-08-03 10:18:45 INFO User has ALL privileges
2016-08-03 10:18:45 INFO binary logs validated on 127.0.0.1:19590
2016-08-03 10:18:45 INFO Restarting replication on 127.0.0.1:19590 to make sure binlog settings apply to replication thread
2016-08-03 10:18:46 INFO Table found. Engine=InnoDB
2016-08-03 10:18:47 INFO Found foreign key on `sakila`.`payment` related to `sakila`.`rental`
2016-08-03 10:18:47 INFO Found foreign key on `sakila`.`rental` related to `sakila`.`rental`
2016-08-03 10:18:47 INFO Found foreign key on `sakila`.`rental` related to `sakila`.`rental`
2016-08-03 10:18:47 INFO Found foreign key on `sakila`.`rental` related to `sakila`.`rental`
2016-08-03 10:18:47 ERROR Found 4 foreign keys related to `sakila`.`rental`. Foreign keys are not supported. Bailing out
2016-08-03 10:18:47 FATAL 2016-08-03 10:18:47 ERROR Found 4 foreign keys related to `sakila`.`rental`. Foreign keys are not supported. Bailing out
```

Related Issue: https://github.com/github/gh-ost/issues/129
2016-08-18 10:07:32 +02:00
Shlomi Noach
3a0ee9b4a5 clarified commented transactional apply 2016-08-17 10:50:41 +02:00
Shlomi Noach
16d76aa299 builder fixes arg type from signed to unsigned based on UnsingedFlags 2016-08-17 06:52:23 +02:00
Shlomi Noach
29d20316ba inspector applies unsigned flags to all migration related columns 2016-08-17 06:51:58 +02:00
Shlomi Noach
f00a9814e6 legacy comments cleanup 2016-08-17 06:51:06 +02:00
Shlomi Noach
4c8edf6372 elaborate error message on applying event data: printing out the error, query and args 2016-08-17 06:50:40 +02:00
Shlomi Noach
7a0e2cfe3f adding UnsignedFlags to ColumnsList 2016-08-17 06:49:20 +02:00
Shlomi Noach
596dce5993 elaborate output on error in apply dml 2016-08-15 15:23:30 +02:00
Shlomi Noach
ac0b788153 rename trust-rbr to assume-rbr 2016-08-15 11:05:51 +02:00
Shlomi Noach
01d48e667c Merge pull request #143 from EasyPost/issue_142
Thank you
2016-08-15 10:53:36 +02:00
Shlomi Noach
1995be2b3f accepting --trust-rbr
- avoiding need to restart replication
- in turn avoiding need for SUPER
2016-08-12 14:26:58 +02:00
Shlomi Noach
8bf07c506f Merge pull request #147 from github/cleanup-socket-file
Cleanup socket file
2016-08-12 11:26:37 +02:00
Shlomi Noach
51111e0aa7 Merge pull request #155 from github/relaxed-log-slave-updates
relaxed check for log_slave_updates
2016-08-12 11:25:18 +02:00
Shlomi Noach
a46022f727 localized function name 2016-08-11 17:37:50 +02:00
Shlomi Noach
66ff5964ed relaxed check for log_slave_updates 2016-08-11 14:49:14 +02:00
Shlomi Noach
95b4f9e440 testing for trigger existence 2016-08-11 14:10:35 +02:00
Shlomi Noach
a1f9c47102 Merge pull request #137 from dveeden/grants_info
Thank you
2016-08-11 09:07:48 +02:00
Shlomi Noach
35ff3f5b7b added log message 2016-08-11 09:03:39 +02:00
Shlomi Noach
dd1ef29dac cleaning up socket file 2016-08-11 09:01:14 +02:00
Shlomi Noach
1418395e99 Merge pull request #124 from dgryski/time-since
Thank you!
2016-08-11 08:50:06 +02:00
James Brown
b2fee9a037 detect IPv6 literals and wrap them in square brackets in GetDBUri 2016-08-05 16:54:26 -07:00
Daniël van Eeden
7786356a45 Fix typo: s/auto_incrmenet/auto_increment/ 2016-08-05 17:46:38 +02:00
Daniël van Eeden
95ba07e01b More user friendly error reporting for privileges
Related Issue: https://github.com/github/gh-ost/issues/136

New output:
```
2016-08-05 11:39:07 DEBUG Privileges: Super: true, REPLICATION SLAVE: false, ALL on *.*: false, ALL on `sakila`.*: true
2016-08-05 11:39:07 ERROR User has insufficient privileges for migration. Needed: SUPER, REPLICATION SLAVE and ALL on `sakila`.*
2016-08-05 11:39:07 FATAL 2016-08-05 11:39:07 ERROR User has insufficient privileges for migration. Needed: SUPER, REPLICATION SLAVE and ALL on `sakila`.*
```
2016-08-05 11:41:36 +02:00
Seth Krasnianski
a94f64b42c [go cmd] Correct spelling for main() comment 2016-08-02 09:22:44 -04:00
Damian Gryski
e02a49449e all: use time.Since() instead of time.Now().Sub
Patch created with:
    gofmt -w -r 'time.Now().Sub(a) -> time.Since(a)' .
2016-08-02 08:38:56 -04:00
Shlomi Noach
46bbea2a32 ETA counting rows, fixed copy time on count 2016-07-29 10:40:23 +02:00
Shlomi Noach
25ce8b0758 status hint parameters using normalized names 2016-07-29 09:20:00 +02:00
Shlomi Noach
edacb8f959 Merge pull request #116 from github/nice-ratio-float
nice-ratio is now float64
2016-07-29 07:16:33 +02:00
Shlomi Noach
be8a023350 nice-ratio is now float64 2016-07-28 14:37:17 +02:00
Shlomi Noach
b99ce969c7 serving socket before counting table rows 2016-07-28 13:01:26 +02:00
Shlomi Noach
b548a6a172 adding human friendly hint re: throttling and binary logs 2016-07-27 10:45:22 +02:00
Shlomi Noach
dbcc0e09c7 status hint shows [set] next to existing flag files 2016-07-27 10:36:24 +02:00
Shlomi Noach
e900dae2e9 More informative information upon control-replicas lagging 2016-07-27 09:59:46 +02:00
Shlomi Noach
b53ee24a1f dynamic replication-lag-query 2016-07-26 14:14:25 +02:00
Shlomi Noach
5d23b72955 Merge pull request #107 from github/throttle-control-replicas
fix to throttle-control-replicas check
2016-07-26 12:13:22 +02:00
Shlomi Noach
7a70c24503 replica-migration cleanup; updating allEventsUpToLockProcessedInjectedFlag 2016-07-26 12:06:20 +02:00
Shlomi Noach
6dbf5c31a2 resolved conflict 2016-07-26 11:57:01 +02:00
Shlomi Noach
034ea7646a fix to throttle-control-replicas check 2016-07-26 11:51:24 +02:00
Shlomi Noach
a75912d1a4 Merge pull request #105 from github/credentials-env-variables
config file supports environment variables
2016-07-26 09:28:23 +02:00
Shlomi Noach
4774b67ffd config file supports environment variables 2016-07-25 15:46:37 +02:00
Shlomi Noach
1d77425fbe capped streamer retries 2016-07-25 15:17:30 +02:00
Shlomi Noach
74804559c8 supporting --initially-drop-socket-file
- by default gh-ost will not delete an existing socket file
  and thus, will fail running if socket file exists. This is the desired behavior.
- The flag --initially-drop-socket-file indicates we take responsibility and wish gh-ost to delete this file on startup
2016-07-22 17:34:18 +02:00
Shlomi Noach
ef59a866d8 Removed legacy 'safe cut-over'
Now that we have the atomic cut-over, the former is redundant
2016-07-16 08:12:19 -06:00
Shlomi Noach
8e46b4ceea max-lag-millis is dynamicly controllable 2016-07-13 09:44:00 +02:00
Shlomi Noach
8217536898 supporting --cut-over-lock-timeout-seconds 2016-07-08 10:14:58 +02:00
Shlomi Noach
c116d84acb added nice-ratio 2016-07-04 14:29:09 +02:00