Shlomi Noach
dcd2f36c88
removed --time-zone flag
2016-10-14 12:53:23 +02:00
Shlomi Noach
9ac468c97e
failing on DATETIME-to-TIMESTAMP conversion within chosen key
2016-10-14 09:33:53 +02:00
Shlomi Noach
11ccf09412
grammar
2016-10-14 09:07:30 +02:00
Shlomi Noach
e6d317b579
fixed inferred usage for --critical-load
2016-10-14 09:05:54 +02:00
Shlomi Noach
661b7ff0c1
added test to sql/types
2016-10-13 13:13:00 +02:00
Shlomi Noach
882959ba83
column is type-aware; is timezone-conversion aware
...
also refactored some functions
2016-10-13 13:12:33 +02:00
Shlomi Noach
a637ed1b50
adapted builder tests
2016-10-13 13:11:35 +02:00
Shlomi Noach
6fbf337051
builder is smarter and can use convert_tz where it notices DATETIME->TIMESTAMP conversion
2016-10-13 13:11:18 +02:00
Shlomi Noach
0981f3f7d3
fix error message placeholder
2016-10-13 13:10:16 +02:00
Shlomi Noach
84feae052f
inspector notes down TIMESTAMP and DATETIME types; notices in particular the conversion from DATETIME to TIMESTAMP
2016-10-13 13:09:58 +02:00
Shlomi Noach
d80694bdac
no need for Inspector timezone; it is irrelevant
2016-10-13 13:08:41 +02:00
Shlomi Noach
f988309800
no need for Inspector timezone; it is irrelevant
2016-10-13 13:08:23 +02:00
Shlomi Noach
ac304def4d
applier always uses UTC
2016-10-13 13:08:02 +02:00
Shlomi Noach
184643157b
Merge branch 'master' into tz-a-different-approach
2016-10-12 08:33:19 +02:00
Shlomi Noach
c11c611755
Merge pull request #260 from github/hooks-nil-check
...
safe access to applier/inspector hostnames for hooks
2016-10-12 08:16:50 +02:00
Shlomi Noach
c399119139
Merge pull request #263 from github/implied-key-fix
...
assume-master-host now applies ImpliedKey
2016-10-12 08:16:38 +02:00
Shlomi Noach
4d903d0119
Merge pull request #264 from github/discard-foreign-keys
...
Discard foreign keys
2016-10-12 08:16:27 +02:00
Shlomi Noach
5d92da4a74
Merge branch 'master' into tz-a-different-approach
2016-10-11 17:17:42 +02:00
Shlomi Noach
8bd36dcb01
Merge branch 'master' into implied-key-fix
2016-10-11 17:17:29 +02:00
Shlomi Noach
c1a6773c02
better handling of --assume-master-host
...
separated logic and not even attempting to crawl topology
2016-10-11 16:42:19 +02:00
Shlomi Noach
dbf50afbc7
reading time_zone settings for Inspector and Applier separately.
...
--time-zone overrides both of them, if given
2016-10-11 16:00:26 +02:00
Shlomi Noach
5de8a010df
restoring original row_event code; tampering with applier time_zone based on global timze_zone
2016-10-10 22:33:54 +02:00
Shlomi Noach
15f4ddfd8a
support for --critical-load-interval-millis
...
this optional flag gives --critical-load a second chance. When configured to positive value, meeting with critical-load spawns a timer. When this timer expires a second check for critical-load is made; if still met, gh-ost bails out.
By default the interval is zero, in which case gh-ost bails out immediately on meeting critical load.
2016-10-10 13:21:01 +02:00
Shlomi Noach
a4d566e71c
counting child vs parent sie foreign keys on table,
...
parent-side foreign keys cannot be discarded
2016-10-10 12:28:56 +02:00
Shlomi Noach
6750959e1a
configurable time-zone, native time parsing
2016-10-10 11:39:57 +02:00
Shlomi Noach
0c35a811f7
setting time_zoe='+00:00' on rowcopy
2016-10-08 11:06:27 +02:00
Shlomi Noach
1f65473e69
support for --discard-foreign-keys
...
This flag makes migration silently and happily discard any existing foreign keys on migrated table. This is useful for intentional dropping of foreign keys, as gh-ost does not otherwise have support for foreign key migration.
At some time in the future gh-ost may support foreign key migration, at which time this flag will be removed
2016-10-07 10:20:50 +02:00
Shlomi Noach
ef04fa49f5
assume-master-host now applied ImpliedKey
2016-10-06 12:00:34 +02:00
Shlomi Noach
72f63d3042
safe access to applier/inspector hostnames for hooks
2016-10-04 21:18:44 +02:00
Shlomi Noach
51f0c0d957
fix arg conversion in BuildDMLUpdateQuery
2016-09-27 13:08:31 +02:00
Paulo Bittencourt
2f80c9d424
Used the source column definition to fetch and convert insert values
...
When processing binlog insert statements, we want to use `sharedColumns`
to decide which values to fetch and convert from the insert DML event.
We only want to `mappedShareColumns` to define the column names in the
`replace into ...` statement.
2016-09-14 11:28:30 -04:00
Shlomi Noach
a7627091a7
Merge branch 'master' into named-cut-over
2016-09-13 05:25:16 -07:00
Shlomi Noach
e5e0444cc6
supporting --force-named-cut-over
...
- when given, user _must_ specify table name
and of course table name must match migrated table
2016-09-12 19:17:36 +02:00
Shlomi Noach
0a8be1dd22
excplicitly breaking on NoPrintStatusRule
2016-09-12 17:39:56 +02:00
Shlomi Noach
1c6f828091
refactored server command into server.go
...
- added support for cut-over=<tablename>
- refactored more code into context
2016-09-12 12:38:14 +02:00
Shlomi Noach
16fc19b157
rowcount progress at 100% when row-copy completes
2016-09-12 10:25:55 +02:00
Shlomi Noach
f4693274fb
Extracted and generalized encoding logic to encoding.go
2016-09-08 09:37:22 +02:00
Shlomi Noach
791d963ea0
Character set recognition and manipulation
...
- Identifying textual characters sets; converting into specific type when applying dml events
- Refactored `ColumnsList`: introducing `Column` type
- Refactored `unsigned` handling, as part of `Column`
- `Column` type supports `convertArg()`: converting value of argument according to column data type
- DB URI attempts `utf8mb4,utf8,latin1` charsets in that order (first one to be recognized wins)
- Local tests filter by pattern
- Local tests append table schema on failure
- Local tests do not have postpone flag file
- Added character set local tests: `utf8`, `utf8mb4`, `latin1`
2016-09-07 14:24:11 +02:00
Shlomi Noach
88f2af8111
support for --assume-master-host, master-master/tungsten
2016-09-02 13:09:18 +02:00
Shlomi Noach
96f108d3b4
Merge pull request #221 from twotwotwo/override-applier-host
...
Add -override-applier-host for use with -allow-master-master
2016-09-02 11:32:04 +02:00
Shlomi Noach
a58e80387e
More environment variables in hooks
2016-09-02 10:48:29 +02:00
Shlomi Noach
75d225353f
Merge pull request #220 from Wattpad/exit-on-hook-replication-stop-failure
...
Fail operation if onStopReplication hook fails
2016-09-02 09:39:43 +02:00
randall
82110fcfcf
Add -override-applier-host for use with -allow-master-master
...
for configurations where writes are meant to go to one master, but gh-ost can't automatically determine which
2016-09-01 20:29:26 -07:00
Paulo Bittencourt
e3662f2398
Fail operation if onStopReplication hook fails
2016-09-01 15:58:20 -04:00
Shlomi Noach
c562df42cd
status: State and ETA decoupling
2016-09-01 10:51:40 +02:00
Shlomi Noach
904215e286
Merge pull request #204 from github/reduce-minimum-max-lag
...
Reduce minimum maxLagMillisecondsThrottleThreshold to 100ms
2016-08-31 09:29:16 +02:00
Shlomi Noach
aef56c55f7
indicating 100% when rowcopy is complete
2016-08-30 17:02:29 +02:00
Shlomi Noach
b2c71931c6
refactored all throttling code into throttler.so
2016-08-30 12:25:45 +02:00
Shlomi Noach
23357d0643
WIP: decoupling general throttling from throttle logic
2016-08-30 11:32:17 +02:00
Shlomi Noach
75b2542f26
Merge branch 'master' into reduce-minimum-max-lag
2016-08-30 09:47:33 +02:00
Shlomi Noach
1c3699cc64
fixed import path
2016-08-30 09:45:02 +02:00
Shlomi Noach
2afb86b9e4
support for millisecond throttling
...
- `--max-lag-millis` is at least `100ms`
- `--heartbeat-interval-millis` introduced; defaults `500ms`, can range `100ms` - `1s`
- Control replicas lag calculated asynchronously to throttle test
- aggressive when `max-lag-millis < 1000` and when `replication-lag-query` is given
2016-08-30 09:41:59 +02:00
Shlomi Noach
176aabead0
Merge pull request #190 from github/hooks
...
WIP: Hooks
2016-08-30 09:01:19 +02:00
Shlomi Noach
c0936ed728
clarified privileges requirements
2016-08-29 12:39:31 +02:00
Shlomi Noach
6dfa4873c2
removed excessive argument
2016-08-29 10:44:43 +02:00
Shlomi Noach
6e5db089c8
supporting onRowCountComplete hook
2016-08-29 09:58:31 +02:00
Jonah Berquist
10b222bc7b
Reduce minimum maxLagMillisecondsThrottleThreshold to 100ms
2016-08-26 16:44:40 -07:00
Shlomi Noach
c70f405d06
Merge branch 'master' into hooks
2016-08-26 08:39:02 +02:00
Shlomi Noach
4f9c1448d1
Merge pull request #200 from github/support-5-5
...
fixed 5.5 binlog_row_image default value
2016-08-26 08:29:33 +02:00
Shlomi Noach
2b595b15f2
Merge pull request #196 from github/concurrent-rowcount
...
concurrent row-count
2016-08-26 08:29:01 +02:00
Shlomi Noach
63a0e5cf2c
fixed 5.5 binlog_row_image default value
2016-08-25 16:49:25 +02:00
Shlomi Noach
ef386add5c
hook output and err are written to gh-ost's stderr
2016-08-25 16:08:49 +02:00
Shlomi Noach
2c9d71570f
logging intented hook invocation
2016-08-25 14:42:49 +02:00
Shlomi Noach
cbdec6bd39
supporting --hooks-path and --hooks-hint
2016-08-25 13:58:36 +02:00
Shlomi Noach
ecd33969bf
added onRowCountComplete(); supporting elapsedSeconds in on-status
2016-08-25 13:55:22 +02:00
Shlomi Noach
fad6743150
added hooks hint message
2016-08-25 13:54:42 +02:00
Shlomi Noach
b064174ab4
added elapsedSeconds to status hook
2016-08-25 13:54:21 +02:00
Shlomi Noach
cb1a7e2805
merged master
2016-08-25 12:32:03 +02:00
Shlomi Noach
c7d88499af
Merge branch 'master' into row-copy-complete
2016-08-25 10:15:32 +02:00
Shlomi Noach
c7edd1ef84
Merge branch 'master' into concurrent-rowcount
2016-08-25 09:44:12 +02:00
Shlomi Noach
1773f338c2
keeping track of delta rows
...
on concurrent count(*) this means we re-apply delta onto new estimate
2016-08-24 12:16:34 +02:00
Shlomi Noach
553f4c8d13
concurrent row-count
2016-08-24 11:39:44 +02:00
Shlomi Noach
4c972184a8
Merge branch 'master' into sql-mode-strict
2016-08-24 09:34:00 +02:00
Shlomi Noach
56fd82a824
Merge pull request #174 from Wattpad/test-on-replica-manual-replication-control
...
outstanding. Thank you!
2016-08-24 09:12:21 +02:00
Paulo Bittencourt
6b21ade6d0
Check for --test-on-replica-skip-replica-stop in cutOver method
2016-08-23 18:34:10 -04:00
Shlomi Noach
56d09c4105
avoiding writing rows when rowcopy complete
2016-08-23 14:26:47 +02:00
Shlomi Noach
8b76d0e75b
DML write sets sql_mode to STRICT ALL TABLES
2016-08-23 11:58:52 +02:00
Shlomi Noach
a1e191078a
rename 'about'->'before'
2016-08-23 11:40:32 +02:00
Shlomi Noach
1c2a77ef95
hook names; added on-stop-replication hook
2016-08-23 11:35:48 +02:00
Shlomi Noach
1021a83ac0
Merge pull request #189 from dveeden/feedback_on_wait
...
thank you
2016-08-23 09:50:50 +02:00
Daniël van Eeden
d8cfd49e2c
Message about waiting should be INFO not DEBUG
2016-08-23 09:41:07 +02:00
Shlomi Noach
972728cf40
added onStatus hook
2016-08-22 16:24:41 +02:00
Shlomi Noach
b63cc3e75e
fix INSERT DML handling on renamed column
2016-08-22 16:00:15 +02:00
Shlomi Noach
9cf4819a98
Merge branch 'master' into fix-rename
...
Wish to incorporate important time_zone fix
2016-08-22 11:54:52 +02:00
Shlomi Noach
1376f0af23
fixed UPDATE dml on renamed column
2016-08-22 08:49:27 +02:00
Shlomi Noach
fc954562a3
Merge pull request #175 from github/apply-dml-tz
...
setting time_zone on DML apply
2016-08-22 08:01:50 +02:00
Shlomi Noach
6acbe7e3ae
detecting and executing hooks
2016-08-20 08:24:20 +02:00
Paulo Bittencourt
2e43718ef3
Add --test-on-replica-skip-replica-stop flag
2016-08-19 17:34:08 -04:00
Shlomi Noach
cdf393a30e
initial support for hooks
2016-08-19 14:52:49 +02:00
Shlomi Noach
d8e30fcd85
fixed sup printing heuristic
2016-08-19 09:41:25 +02:00
Shlomi Noach
9752179723
interactive command: sup
2016-08-19 09:16:17 +02:00
Shlomi Noach
6d80340e4f
setting time_zone on DML apply
2016-08-19 09:06:00 +02:00
Shlomi Noach
e6a02d81e0
Merge pull request #170 from github/nice-ratio-doc-clarification
...
clarifying meaning of sleep-ratio
2016-08-19 08:27:22 +02:00
Shlomi Noach
1e88dbd04a
Merge pull request #171 from github/empty-table-progress
...
progress is 100% when 0/0 rows copied
2016-08-19 08:27:04 +02:00
Shlomi Noach
4a7df07f64
Merge pull request #173 from github/accept-mysql-prompt-config
...
relaxed config scanner mode
2016-08-19 08:26:13 +02:00
Shlomi Noach
41d43933c8
Merge pull request #172 from github/transactional-apply-dml
...
DML write wrapped in transaction
2016-08-19 08:25:52 +02:00
Paulo Bittencourt
a62f9e0754
Add --test-on-replica-manual-replication-control flag
...
This will wait indefinitely for the replication status to change.
This allows us to run test schema changes in RDS without needing
custom RDS commands in gh-ost.
2016-08-18 11:53:25 -04:00
Shlomi Noach
00369d7e5d
relaxed config scanner mode
...
- does not fail on MySQL 'prompt' config
2016-08-18 13:58:38 +02:00
Shlomi Noach
88eb2d6ee1
Merge pull request #130 from dveeden/foreign_key_info
...
Give more info about foreign keys
2016-08-18 13:40:31 +02:00
Shlomi Noach
75e0d12302
simplified error logic; fixed incorrect RowsEstimate handling on error
2016-08-18 13:38:23 +02:00
Shlomi Noach
74593ec010
DML write wrapped in transaction
...
- solving the golang problem: 'sql: converting Exec argument #2 's type: uint64 values with high bit set are not supported'
2016-08-18 13:31:53 +02:00
Shlomi Noach
7e9f578e12
progress is 100% when 0/0 rows copied
2016-08-18 13:20:09 +02:00
Shlomi Noach
5dbd2e1c85
clarifying meaning of sleep-ratio
2016-08-18 13:13:51 +02:00
Shlomi Noach
d9ae2f3942
Merge pull request #156 from github/avoid-restarting-replication
...
accepting `--assume-rbr`, remove `SUPER` requirement
2016-08-18 10:11:35 +02:00
Daniël van Eeden
d3422bd19a
Give more info about foreign keys
...
This helps when the table itself doesn't have foreign keys defined but if there are other tables with foreign keys pointing to the table on which gh-ost runs.
This gives INFO messages for each FK. Note that it now informs the user about `payment` being involved.
```
$ ./gh-ost -database sakila -table rental -alter 'ADD COLUMN ghost_test_001 tinyint DEFAULT NULL' -port 19590 -user msandbox -password msandbox -verbose
2016-08-03 10:18:45 INFO starting gh-ost 1.0.8
2016-08-03 10:18:45 INFO Migrating `sakila`.`rental`
2016-08-03 10:18:45 INFO connection validated on 127.0.0.1:19590
2016-08-03 10:18:45 INFO User has ALL privileges
2016-08-03 10:18:45 INFO binary logs validated on 127.0.0.1:19590
2016-08-03 10:18:45 INFO Restarting replication on 127.0.0.1:19590 to make sure binlog settings apply to replication thread
2016-08-03 10:18:46 INFO Table found. Engine=InnoDB
2016-08-03 10:18:47 INFO Found foreign key on `sakila`.`payment` related to `sakila`.`rental`
2016-08-03 10:18:47 INFO Found foreign key on `sakila`.`rental` related to `sakila`.`rental`
2016-08-03 10:18:47 INFO Found foreign key on `sakila`.`rental` related to `sakila`.`rental`
2016-08-03 10:18:47 INFO Found foreign key on `sakila`.`rental` related to `sakila`.`rental`
2016-08-03 10:18:47 ERROR Found 4 foreign keys related to `sakila`.`rental`. Foreign keys are not supported. Bailing out
2016-08-03 10:18:47 FATAL 2016-08-03 10:18:47 ERROR Found 4 foreign keys related to `sakila`.`rental`. Foreign keys are not supported. Bailing out
```
Related Issue: https://github.com/github/gh-ost/issues/129
2016-08-18 10:07:32 +02:00
Shlomi Noach
3a0ee9b4a5
clarified commented transactional apply
2016-08-17 10:50:41 +02:00
Shlomi Noach
16d76aa299
builder fixes arg type from signed to unsigned based on UnsingedFlags
2016-08-17 06:52:23 +02:00
Shlomi Noach
29d20316ba
inspector applies unsigned flags to all migration related columns
2016-08-17 06:51:58 +02:00
Shlomi Noach
f00a9814e6
legacy comments cleanup
2016-08-17 06:51:06 +02:00
Shlomi Noach
4c8edf6372
elaborate error message on applying event data: printing out the error, query and args
2016-08-17 06:50:40 +02:00
Shlomi Noach
7a0e2cfe3f
adding UnsignedFlags to ColumnsList
2016-08-17 06:49:20 +02:00
Shlomi Noach
596dce5993
elaborate output on error in apply dml
2016-08-15 15:23:30 +02:00
Shlomi Noach
ac0b788153
rename trust-rbr to assume-rbr
2016-08-15 11:05:51 +02:00
Shlomi Noach
01d48e667c
Merge pull request #143 from EasyPost/issue_142
...
Thank you
2016-08-15 10:53:36 +02:00
Shlomi Noach
1995be2b3f
accepting --trust-rbr
...
- avoiding need to restart replication
- in turn avoiding need for SUPER
2016-08-12 14:26:58 +02:00
Shlomi Noach
8bf07c506f
Merge pull request #147 from github/cleanup-socket-file
...
Cleanup socket file
2016-08-12 11:26:37 +02:00
Shlomi Noach
51111e0aa7
Merge pull request #155 from github/relaxed-log-slave-updates
...
relaxed check for log_slave_updates
2016-08-12 11:25:18 +02:00
Shlomi Noach
a46022f727
localized function name
2016-08-11 17:37:50 +02:00
Shlomi Noach
66ff5964ed
relaxed check for log_slave_updates
2016-08-11 14:49:14 +02:00
Shlomi Noach
95b4f9e440
testing for trigger existence
2016-08-11 14:10:35 +02:00
Shlomi Noach
a1f9c47102
Merge pull request #137 from dveeden/grants_info
...
Thank you
2016-08-11 09:07:48 +02:00
Shlomi Noach
35ff3f5b7b
added log message
2016-08-11 09:03:39 +02:00
Shlomi Noach
dd1ef29dac
cleaning up socket file
2016-08-11 09:01:14 +02:00
Shlomi Noach
1418395e99
Merge pull request #124 from dgryski/time-since
...
Thank you!
2016-08-11 08:50:06 +02:00
James Brown
b2fee9a037
detect IPv6 literals and wrap them in square brackets in GetDBUri
2016-08-05 16:54:26 -07:00
Daniël van Eeden
7786356a45
Fix typo: s/auto_incrmenet/auto_increment/
2016-08-05 17:46:38 +02:00
Daniël van Eeden
95ba07e01b
More user friendly error reporting for privileges
...
Related Issue: https://github.com/github/gh-ost/issues/136
New output:
```
2016-08-05 11:39:07 DEBUG Privileges: Super: true, REPLICATION SLAVE: false, ALL on *.*: false, ALL on `sakila`.*: true
2016-08-05 11:39:07 ERROR User has insufficient privileges for migration. Needed: SUPER, REPLICATION SLAVE and ALL on `sakila`.*
2016-08-05 11:39:07 FATAL 2016-08-05 11:39:07 ERROR User has insufficient privileges for migration. Needed: SUPER, REPLICATION SLAVE and ALL on `sakila`.*
```
2016-08-05 11:41:36 +02:00
Seth Krasnianski
a94f64b42c
[go cmd] Correct spelling for main() comment
2016-08-02 09:22:44 -04:00
Damian Gryski
e02a49449e
all: use time.Since() instead of time.Now().Sub
...
Patch created with:
gofmt -w -r 'time.Now().Sub(a) -> time.Since(a)' .
2016-08-02 08:38:56 -04:00
Shlomi Noach
46bbea2a32
ETA counting rows, fixed copy time on count
2016-07-29 10:40:23 +02:00
Shlomi Noach
25ce8b0758
status hint parameters using normalized names
2016-07-29 09:20:00 +02:00
Shlomi Noach
edacb8f959
Merge pull request #116 from github/nice-ratio-float
...
nice-ratio is now float64
2016-07-29 07:16:33 +02:00
Shlomi Noach
be8a023350
nice-ratio is now float64
2016-07-28 14:37:17 +02:00
Shlomi Noach
b99ce969c7
serving socket before counting table rows
2016-07-28 13:01:26 +02:00
Shlomi Noach
b548a6a172
adding human friendly hint re: throttling and binary logs
2016-07-27 10:45:22 +02:00
Shlomi Noach
dbcc0e09c7
status hint shows [set] next to existing flag files
2016-07-27 10:36:24 +02:00
Shlomi Noach
e900dae2e9
More informative information upon control-replicas lagging
2016-07-27 09:59:46 +02:00
Shlomi Noach
b53ee24a1f
dynamic replication-lag-query
2016-07-26 14:14:25 +02:00
Shlomi Noach
5d23b72955
Merge pull request #107 from github/throttle-control-replicas
...
fix to throttle-control-replicas check
2016-07-26 12:13:22 +02:00
Shlomi Noach
7a70c24503
replica-migration cleanup; updating allEventsUpToLockProcessedInjectedFlag
2016-07-26 12:06:20 +02:00
Shlomi Noach
6dbf5c31a2
resolved conflict
2016-07-26 11:57:01 +02:00
Shlomi Noach
034ea7646a
fix to throttle-control-replicas check
2016-07-26 11:51:24 +02:00
Shlomi Noach
a75912d1a4
Merge pull request #105 from github/credentials-env-variables
...
config file supports environment variables
2016-07-26 09:28:23 +02:00
Shlomi Noach
4774b67ffd
config file supports environment variables
2016-07-25 15:46:37 +02:00
Shlomi Noach
1d77425fbe
capped streamer retries
2016-07-25 15:17:30 +02:00
Shlomi Noach
74804559c8
supporting --initially-drop-socket-file
...
- by default gh-ost will not delete an existing socket file
and thus, will fail running if socket file exists. This is the desired behavior.
- The flag --initially-drop-socket-file indicates we take responsibility and wish gh-ost to delete this file on startup
2016-07-22 17:34:18 +02:00
Shlomi Noach
ef59a866d8
Removed legacy 'safe cut-over'
...
Now that we have the atomic cut-over, the former is redundant
2016-07-16 08:12:19 -06:00
Shlomi Noach
8e46b4ceea
max-lag-millis is dynamicly controllable
2016-07-13 09:44:00 +02:00
Shlomi Noach
8217536898
supporting --cut-over-lock-timeout-seconds
2016-07-08 10:14:58 +02:00
Shlomi Noach
c116d84acb
added nice-ratio
2016-07-04 14:29:09 +02:00