octoleo/gh-ost
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
559 Commits 51 Branches 51 Tags 24 MiB
b696106cb6
Commit Graph

250 Commits

Author SHA1 Message Date
Shlomi Noach
904215e286 Merge pull request #204 from github/reduce-minimum-max-lag 
Reduce minimum maxLagMillisecondsThrottleThreshold to 100ms
 2016-08-31 09:29:16 +02:00
Shlomi Noach
aef56c55f7 indicating 100% when rowcopy is complete 2016-08-30 17:02:29 +02:00
Shlomi Noach
b2c71931c6 refactored all throttling code into throttler.so 2016-08-30 12:25:45 +02:00
Shlomi Noach
23357d0643 WIP: decoupling general throttling from throttle logic 2016-08-30 11:32:17 +02:00
Shlomi Noach
75b2542f26 Merge branch 'master' into reduce-minimum-max-lag 2016-08-30 09:47:33 +02:00
Shlomi Noach
1c3699cc64 fixed import path 2016-08-30 09:45:02 +02:00
Shlomi Noach
2afb86b9e4 support for millisecond throttling 
- `--max-lag-millis` is at least `100ms`
- `--heartbeat-interval-millis` introduced; defaults `500ms`, can range `100ms` - `1s`
- Control replicas lag calculated asynchronously to throttle test
  - aggressive when `max-lag-millis < 1000` and when `replication-lag-query` is given
 2016-08-30 09:41:59 +02:00
Shlomi Noach
176aabead0 Merge pull request #190 from github/hooks 
WIP: Hooks
 2016-08-30 09:01:19 +02:00
Shlomi Noach
c0936ed728 clarified privileges requirements 2016-08-29 12:39:31 +02:00
Shlomi Noach
6dfa4873c2 removed excessive argument 2016-08-29 10:44:43 +02:00
Shlomi Noach
6e5db089c8 supporting onRowCountComplete hook 2016-08-29 09:58:31 +02:00
Jonah Berquist
10b222bc7b Reduce minimum maxLagMillisecondsThrottleThreshold to 100ms 2016-08-26 16:44:40 -07:00
Shlomi Noach
c70f405d06 Merge branch 'master' into hooks 2016-08-26 08:39:02 +02:00
Shlomi Noach
4f9c1448d1 Merge pull request #200 from github/support-5-5 
fixed 5.5 binlog_row_image default value
 2016-08-26 08:29:33 +02:00
Shlomi Noach
2b595b15f2 Merge pull request #196 from github/concurrent-rowcount 
concurrent row-count
 2016-08-26 08:29:01 +02:00
Shlomi Noach
63a0e5cf2c fixed 5.5 binlog_row_image default value 2016-08-25 16:49:25 +02:00
Shlomi Noach
ef386add5c hook output and err are written to gh-ost's stderr 2016-08-25 16:08:49 +02:00
Shlomi Noach
2c9d71570f logging intented hook invocation 2016-08-25 14:42:49 +02:00
Shlomi Noach
cbdec6bd39 supporting --hooks-path and --hooks-hint 2016-08-25 13:58:36 +02:00
Shlomi Noach
ecd33969bf added onRowCountComplete(); supporting elapsedSeconds in on-status 2016-08-25 13:55:22 +02:00
Shlomi Noach
fad6743150 added hooks hint message 2016-08-25 13:54:42 +02:00
Shlomi Noach
b064174ab4 added elapsedSeconds to status hook 2016-08-25 13:54:21 +02:00
Shlomi Noach
cb1a7e2805 merged master 2016-08-25 12:32:03 +02:00
Shlomi Noach
c7d88499af Merge branch 'master' into row-copy-complete 2016-08-25 10:15:32 +02:00
Shlomi Noach
c7edd1ef84 Merge branch 'master' into concurrent-rowcount 2016-08-25 09:44:12 +02:00
Shlomi Noach
1773f338c2 keeping track of delta rows 
on concurrent count(*) this means we re-apply delta onto new estimate
 2016-08-24 12:16:34 +02:00
Shlomi Noach
553f4c8d13 concurrent row-count 2016-08-24 11:39:44 +02:00
Shlomi Noach
4c972184a8 Merge branch 'master' into sql-mode-strict 2016-08-24 09:34:00 +02:00
Shlomi Noach
56fd82a824 Merge pull request #174 from Wattpad/test-on-replica-manual-replication-control 
outstanding. Thank you!
 2016-08-24 09:12:21 +02:00
Paulo Bittencourt
6b21ade6d0 Check for --test-on-replica-skip-replica-stop in cutOver method 2016-08-23 18:34:10 -04:00
Shlomi Noach
56d09c4105 avoiding writing rows when rowcopy complete 2016-08-23 14:26:47 +02:00
Shlomi Noach
8b76d0e75b DML write sets sql_mode to STRICT ALL TABLES 2016-08-23 11:58:52 +02:00
Shlomi Noach
a1e191078a rename 'about'->'before' 2016-08-23 11:40:32 +02:00
Shlomi Noach
1c2a77ef95 hook names; added on-stop-replication hook 2016-08-23 11:35:48 +02:00
Shlomi Noach
1021a83ac0 Merge pull request #189 from dveeden/feedback_on_wait 
thank you
 2016-08-23 09:50:50 +02:00
Daniël van Eeden
d8cfd49e2c Message about waiting should be INFO not DEBUG 2016-08-23 09:41:07 +02:00
Shlomi Noach
972728cf40 added onStatus hook 2016-08-22 16:24:41 +02:00
Shlomi Noach
b63cc3e75e fix INSERT DML handling on renamed column 2016-08-22 16:00:15 +02:00
Shlomi Noach
9cf4819a98 Merge branch 'master' into fix-rename 
Wish to incorporate important time_zone fix
 2016-08-22 11:54:52 +02:00
Shlomi Noach
1376f0af23 fixed UPDATE dml on renamed column 2016-08-22 08:49:27 +02:00
Shlomi Noach
fc954562a3 Merge pull request #175 from github/apply-dml-tz 
setting time_zone on DML apply
 2016-08-22 08:01:50 +02:00
Shlomi Noach
6acbe7e3ae detecting and executing hooks 2016-08-20 08:24:20 +02:00
Paulo Bittencourt
2e43718ef3 Add --test-on-replica-skip-replica-stop flag 2016-08-19 17:34:08 -04:00
Shlomi Noach
cdf393a30e initial support for hooks 2016-08-19 14:52:49 +02:00
Shlomi Noach
d8e30fcd85 fixed sup printing heuristic 2016-08-19 09:41:25 +02:00
Shlomi Noach
9752179723 interactive command: sup 2016-08-19 09:16:17 +02:00
Shlomi Noach
6d80340e4f setting time_zone on DML apply 2016-08-19 09:06:00 +02:00
Shlomi Noach
e6a02d81e0 Merge pull request #170 from github/nice-ratio-doc-clarification 
clarifying meaning of sleep-ratio
 2016-08-19 08:27:22 +02:00
Shlomi Noach
1e88dbd04a Merge pull request #171 from github/empty-table-progress 
progress is 100% when 0/0 rows copied
 2016-08-19 08:27:04 +02:00
Shlomi Noach
4a7df07f64 Merge pull request #173 from github/accept-mysql-prompt-config 
relaxed config scanner mode
 2016-08-19 08:26:13 +02:00
Shlomi Noach
41d43933c8 Merge pull request #172 from github/transactional-apply-dml 
DML write wrapped in transaction
 2016-08-19 08:25:52 +02:00
Paulo Bittencourt
a62f9e0754 Add --test-on-replica-manual-replication-control flag 
This will wait indefinitely for the replication status to change.
This allows us to run test schema changes in RDS without needing
custom RDS commands in gh-ost.
 2016-08-18 11:53:25 -04:00
Shlomi Noach
00369d7e5d relaxed config scanner mode 
- does not fail on MySQL 'prompt' config
 2016-08-18 13:58:38 +02:00
Shlomi Noach
88eb2d6ee1 Merge pull request #130 from dveeden/foreign_key_info 
Give more info about foreign keys
 2016-08-18 13:40:31 +02:00
Shlomi Noach
75e0d12302 simplified error logic; fixed incorrect RowsEstimate handling on error 2016-08-18 13:38:23 +02:00
Shlomi Noach
74593ec010 DML write wrapped in transaction 
- solving the golang problem: 'sql: converting Exec argument #2's type: uint64 values with high bit set are not supported'
 2016-08-18 13:31:53 +02:00
Shlomi Noach
7e9f578e12 progress is 100% when 0/0 rows copied 2016-08-18 13:20:09 +02:00
Shlomi Noach
5dbd2e1c85 clarifying meaning of sleep-ratio 2016-08-18 13:13:51 +02:00
Shlomi Noach
d9ae2f3942 Merge pull request #156 from github/avoid-restarting-replication 
accepting `--assume-rbr`, remove `SUPER` requirement
 2016-08-18 10:11:35 +02:00
Daniël van Eeden
d3422bd19a Give more info about foreign keys 
This helps when the table itself doesn't have foreign keys defined but if there are other tables with foreign keys pointing to the table on which gh-ost runs.

This gives INFO messages for each FK. Note that it now informs the user about `payment` being involved.
```
$ ./gh-ost -database sakila -table rental -alter 'ADD COLUMN ghost_test_001 tinyint DEFAULT NULL' -port 19590 -user msandbox -password msandbox -verbose
2016-08-03 10:18:45 INFO starting gh-ost 1.0.8
2016-08-03 10:18:45 INFO Migrating `sakila`.`rental`
2016-08-03 10:18:45 INFO connection validated on 127.0.0.1:19590
2016-08-03 10:18:45 INFO User has ALL privileges
2016-08-03 10:18:45 INFO binary logs validated on 127.0.0.1:19590
2016-08-03 10:18:45 INFO Restarting replication on 127.0.0.1:19590 to make sure binlog settings apply to replication thread
2016-08-03 10:18:46 INFO Table found. Engine=InnoDB
2016-08-03 10:18:47 INFO Found foreign key on `sakila`.`payment` related to `sakila`.`rental`
2016-08-03 10:18:47 INFO Found foreign key on `sakila`.`rental` related to `sakila`.`rental`
2016-08-03 10:18:47 INFO Found foreign key on `sakila`.`rental` related to `sakila`.`rental`
2016-08-03 10:18:47 INFO Found foreign key on `sakila`.`rental` related to `sakila`.`rental`
2016-08-03 10:18:47 ERROR Found 4 foreign keys related to `sakila`.`rental`. Foreign keys are not supported. Bailing out
2016-08-03 10:18:47 FATAL 2016-08-03 10:18:47 ERROR Found 4 foreign keys related to `sakila`.`rental`. Foreign keys are not supported. Bailing out
```

Related Issue: https://github.com/github/gh-ost/issues/129
 2016-08-18 10:07:32 +02:00
Shlomi Noach
3a0ee9b4a5 clarified commented transactional apply 2016-08-17 10:50:41 +02:00
Shlomi Noach
16d76aa299 builder fixes arg type from signed to unsigned based on UnsingedFlags 2016-08-17 06:52:23 +02:00
Shlomi Noach
29d20316ba inspector applies unsigned flags to all migration related columns 2016-08-17 06:51:58 +02:00
Shlomi Noach
f00a9814e6 legacy comments cleanup 2016-08-17 06:51:06 +02:00
Shlomi Noach
4c8edf6372 elaborate error message on applying event data: printing out the error, query and args 2016-08-17 06:50:40 +02:00
Shlomi Noach
7a0e2cfe3f adding UnsignedFlags to ColumnsList 2016-08-17 06:49:20 +02:00
Shlomi Noach
596dce5993 elaborate output on error in apply dml 2016-08-15 15:23:30 +02:00
Shlomi Noach
ac0b788153 rename trust-rbr to assume-rbr 2016-08-15 11:05:51 +02:00
Shlomi Noach
01d48e667c Merge pull request #143 from EasyPost/issue_142 
Thank you
 2016-08-15 10:53:36 +02:00
Shlomi Noach
1995be2b3f accepting --trust-rbr 
- avoiding need to restart replication
- in turn avoiding need for SUPER
 2016-08-12 14:26:58 +02:00
Shlomi Noach
8bf07c506f Merge pull request #147 from github/cleanup-socket-file 
Cleanup socket file
 2016-08-12 11:26:37 +02:00
Shlomi Noach
51111e0aa7 Merge pull request #155 from github/relaxed-log-slave-updates 
relaxed check for log_slave_updates
 2016-08-12 11:25:18 +02:00
Shlomi Noach
a46022f727 localized function name 2016-08-11 17:37:50 +02:00
Shlomi Noach
66ff5964ed relaxed check for log_slave_updates 2016-08-11 14:49:14 +02:00
Shlomi Noach
95b4f9e440 testing for trigger existence 2016-08-11 14:10:35 +02:00
Shlomi Noach
a1f9c47102 Merge pull request #137 from dveeden/grants_info 
Thank you
 2016-08-11 09:07:48 +02:00
Shlomi Noach
35ff3f5b7b added log message 2016-08-11 09:03:39 +02:00
Shlomi Noach
dd1ef29dac cleaning up socket file 2016-08-11 09:01:14 +02:00
Shlomi Noach
1418395e99 Merge pull request #124 from dgryski/time-since 
Thank you!
 2016-08-11 08:50:06 +02:00
James Brown
b2fee9a037 detect IPv6 literals and wrap them in square brackets in GetDBUri 2016-08-05 16:54:26 -07:00
Daniël van Eeden
7786356a45 Fix typo: s/auto_incrmenet/auto_increment/ 2016-08-05 17:46:38 +02:00
Daniël van Eeden
95ba07e01b More user friendly error reporting for privileges 
Related Issue: https://github.com/github/gh-ost/issues/136

New output:
```
2016-08-05 11:39:07 DEBUG Privileges: Super: true, REPLICATION SLAVE: false, ALL on *.*: false, ALL on `sakila`.*: true
2016-08-05 11:39:07 ERROR User has insufficient privileges for migration. Needed: SUPER, REPLICATION SLAVE and ALL on `sakila`.*
2016-08-05 11:39:07 FATAL 2016-08-05 11:39:07 ERROR User has insufficient privileges for migration. Needed: SUPER, REPLICATION SLAVE and ALL on `sakila`.*
```
 2016-08-05 11:41:36 +02:00
Seth Krasnianski
a94f64b42c [go cmd] Correct spelling for main() comment 2016-08-02 09:22:44 -04:00
Damian Gryski
e02a49449e all: use time.Since() instead of time.Now().Sub 
Patch created with:
    gofmt -w -r 'time.Now().Sub(a) -> time.Since(a)' .
 2016-08-02 08:38:56 -04:00
Shlomi Noach
46bbea2a32 ETA counting rows, fixed copy time on count 2016-07-29 10:40:23 +02:00
Shlomi Noach
25ce8b0758 status hint parameters using normalized names 2016-07-29 09:20:00 +02:00
Shlomi Noach
edacb8f959 Merge pull request #116 from github/nice-ratio-float 
nice-ratio is now float64
 2016-07-29 07:16:33 +02:00
Shlomi Noach
be8a023350 nice-ratio is now float64 2016-07-28 14:37:17 +02:00
Shlomi Noach
b99ce969c7 serving socket before counting table rows 2016-07-28 13:01:26 +02:00
Shlomi Noach
b548a6a172 adding human friendly hint re: throttling and binary logs 2016-07-27 10:45:22 +02:00
Shlomi Noach
dbcc0e09c7 status hint shows [set] next to existing flag files 2016-07-27 10:36:24 +02:00
Shlomi Noach
e900dae2e9 More informative information upon control-replicas lagging 2016-07-27 09:59:46 +02:00
Shlomi Noach
b53ee24a1f dynamic replication-lag-query 2016-07-26 14:14:25 +02:00
Shlomi Noach
5d23b72955 Merge pull request #107 from github/throttle-control-replicas 
fix to throttle-control-replicas check
 2016-07-26 12:13:22 +02:00
Shlomi Noach
7a70c24503 replica-migration cleanup; updating allEventsUpToLockProcessedInjectedFlag 2016-07-26 12:06:20 +02:00
Shlomi Noach
6dbf5c31a2 resolved conflict 2016-07-26 11:57:01 +02:00
Shlomi Noach
034ea7646a fix to throttle-control-replicas check 2016-07-26 11:51:24 +02:00
Shlomi Noach
a75912d1a4 Merge pull request #105 from github/credentials-env-variables 
config file supports environment variables
 2016-07-26 09:28:23 +02:00
Shlomi Noach
4774b67ffd config file supports environment variables 2016-07-25 15:46:37 +02:00
Shlomi Noach
1d77425fbe capped streamer retries 2016-07-25 15:17:30 +02:00