Commit Graph

55 Commits

Author SHA1 Message Date
Shlomi Noach
bf92eec214 validating table structure on applier and migrator
- reading column list on applier
- comparing original table on applier and migrator, expecting exact column list
- or else bailing out
2016-10-20 11:29:30 +02:00
Shlomi Noach
6c7ab1a3e5 updated version 2016-10-13 13:14:59 +02:00
Shlomi Noach
b38af6c29e updated version 2016-10-12 08:33:49 +02:00
Shlomi Noach
5d312a2bfb documenting --discard-foreign-keys flag 2016-10-07 10:23:52 +02:00
Shlomi Noach
7517d48776 updated version 2016-09-13 05:35:43 -07:00
Shlomi Noach
1c6f828091 refactored server command into server.go
- added support for cut-over=<tablename>
- refactored more code into context
2016-09-12 12:38:14 +02:00
Shlomi Noach
791d963ea0 Character set recognition and manipulation
- Identifying textual characters sets; converting into specific type when applying dml events
- Refactored `ColumnsList`: introducing `Column` type
- Refactored `unsigned` handling, as part of `Column`
- `Column` type supports `convertArg()`: converting value of argument according to column data type
- DB URI attempts `utf8mb4,utf8,latin1` charsets in that order (first one to be recognized wins)
- Local tests filter by pattern
- Local tests append table schema on failure
- Local tests do not have postpone flag file
- Added character set local tests: `utf8`, `utf8mb4`, `latin1`
2016-09-07 14:24:11 +02:00
Shlomi Noach
a58e80387e More environment variables in hooks 2016-09-02 10:48:29 +02:00
Shlomi Noach
43a3ce8a58 updated version 2016-09-01 10:51:56 +02:00
Shlomi Noach
89a3a261a8 updated version 2016-08-26 08:39:32 +02:00
Shlomi Noach
bfcedef8cf updated version 2016-08-25 12:20:33 +02:00
Shlomi Noach
9dc378feaa updated version 2016-08-22 16:20:11 +02:00
Shlomi Noach
cf5fdb971b updated version 2016-08-22 11:55:33 +02:00
Shlomi Noach
6d80340e4f setting time_zone on DML apply 2016-08-19 09:06:00 +02:00
Shlomi Noach
3a0ee9b4a5 clarified commented transactional apply 2016-08-17 10:50:41 +02:00
Shlomi Noach
98f6ddf00c merged dveeden-build_failfast 2016-08-12 13:04:17 +02:00
Shlomi Noach
95b4f9e440 testing for trigger existence 2016-08-11 14:10:35 +02:00
Daniël van Eeden
d837c6b8e2 Set executable bits on build.sh 2016-08-04 07:13:46 +02:00
Shlomi Noach
8e17880608 upped version 2016-08-01 10:36:17 +02:00
Shlomi Noach
bae4af8c76 updated version 2016-07-31 09:18:50 +02:00
Shlomi Noach
b53ee24a1f dynamic replication-lag-query 2016-07-26 14:14:25 +02:00
Shlomi Noach
6dbf5c31a2 resolved conflict 2016-07-26 11:57:01 +02:00
Shlomi Noach
4774b67ffd config file supports environment variables 2016-07-25 15:46:37 +02:00
Shlomi Noach
74804559c8 supporting --initially-drop-socket-file
- by default gh-ost will not delete an existing socket file
  and thus, will fail running if socket file exists. This is the desired behavior.
- The flag --initially-drop-socket-file indicates we take responsibility and wish gh-ost to delete this file on startup
2016-07-22 17:34:18 +02:00
Shlomi Noach
ef59a866d8 Removed legacy 'safe cut-over'
Now that we have the atomic cut-over, the former is redundant
2016-07-16 08:12:19 -06:00
Shlomi Noach
8e46b4ceea max-lag-millis is dynamicly controllable 2016-07-13 09:44:00 +02:00
Shlomi Noach
8217536898 supporting --cut-over-lock-timeout-seconds 2016-07-08 10:14:58 +02:00
Shlomi Noach
c116d84acb added nice-ratio 2016-07-04 14:29:09 +02:00
Shlomi Noach
0191b2897d an atomic cut-over implementation, as per issue #82 2016-06-27 11:08:06 +02:00
Shlomi Noach
96e8419a35 Solved cut-over stall; change of table names
- Cutover would stall after `lock tables` wait-timeout due do waiting on a channel that would never be written to. This has been identified, reproduced, fixed, confirmed.
- Change of table names. Heres the story:
  - Because were testing this even while `pt-online-schema-change` is being used in production, the `_tbl_old` naming convention makes for a collision.
  - "old" table name is now `_tbl_del`, "del" standing for "delete"
  - ghost table name is now `_tbl_gho`
  - when issuing `--test-on-replica`, we keep the ghost table around, and were also briefly renaming original table to "old". Well this collides with a potentially existing "old" table on master (one that hasnt been dropped yet).
  `--test-on-replica` uses `_tbl_ght` (ghost-test)
  - similar problem with `--execute-on-replica`, and in this case the table doesnt stick around; calling it `_tbl_ghr` (ghost-replica)
  - changelog table is now `_tbl_ghc` (ghost-changelog)
  - To clarify, I dont want to go down the path of creating "old" tables with 2 or 3 or 4 or 5 or infinite leading underscored. I think this is very confusing and actually not operations friendly. Its OK that the migration will fail saying "hey, you ALREADY have an old table here, why dont you take care of it first", rather than create _yet_another_ `____tbl_old` table. Were always confused on which table it actually is that gets migrated, which is safe to `drop`, etc.
- just after rowcopy completing, just before cutover, during cutover: marking as point in time _of interest_ so as to increase logging frequency.
2016-06-21 12:56:01 +02:00
Shlomi Noach
2c8121c070 upped version 2016-06-20 12:09:35 +02:00
Shlomi Noach
23cb8ea7e9 Throttling & critical load
- Added `--throttle-query` param (when returns > 0, throttling applies)
- Added `--critical-load`, similar to `--max-load` but implies panic and quit
- Recoded *-load as `LoadMap`
- More info on *-load throttle/panic
- `printStatus()` now gets printing heuristic. Always shows up on interactive `"status"`
- Fixed `change column` (aka rename) handling with quotes
- Removed legacy `mysqlbinlog` parser code
- Added tests
2016-06-18 21:12:07 +02:00
Shlomi Noach
94f311ec7b supporting --panic-flag-file; when it exists - app panics and exits without cleanup 2016-06-17 11:40:08 +02:00
Shlomi Noach
836d0fe119 Supporting column rename
- Parsing `alter` statement to catch `change old_name new_name ...` statements
- Auto deducing renamed columns
- When suspecting renamed columns, requesting explicit `--approve-renamed-columns` or `--skip-renamed-columns`
- updated tests
2016-06-17 08:03:18 +02:00
Shlomi Noach
1226fa893f updated documentation re: cut-over, test-on-replica, migrate-on-replica 2016-06-15 12:31:38 +02:00
Shlomi Noach
96bc3804eb test-on-replica stops replication completely 2016-06-14 12:50:07 +02:00
Shlomi Noach
97adbf1ff8 - --cut-over no longer mandatory; default to safe
- Removed `CutOverVoluntaryLock` and associated code
- Removed `CutOverUdfWait`
- `RenameTablesRollback()` first attempts an atomic swap
2016-06-14 09:01:06 +02:00
Shlomi Noach
b8c7e046a1 test-on-replica to invoke cut-over swap 2016-06-10 11:15:11 +02:00
Shlomi Noach
087d1dd64d suuporting dynamic reconfiguration of max-load 2016-06-09 11:25:01 +02:00
Shlomi Noach
d30638f1cc updated version 2016-06-07 14:26:15 +02:00
Shlomi Noach
a6c21dcdb0 - --postpone-swap-tables-flag-file renamed to --postpone-cut-over-flag-file
- More `README` documentation
- Added "throttle" documentation
2016-06-07 14:05:25 +02:00
Shlomi Noach
fc00cb2289 adding interactive user commands 2016-06-07 11:59:17 +02:00
Shlomi Noach
bbd19abc9a - requiring --cut-over argument to be two-step|voluntary-lock (will add udf-wait once it is ready)
The idea is that the user is forced to specify the cut-over type they wish to use, given that each type has some drawbacks.
- More data in status hint
- `select count(*)` is deferred till after we validate migration is valid. Also, it is skipped on `--noop`
2016-06-06 12:33:05 +02:00
Shlomi Noach
b73eb5adfc updated version 2016-05-31 21:13:01 +02:00
Shlomi Noach
583d6d3147 accepting SIGHUP. Reloads configuration and marks as point of interest 2016-05-25 12:27:58 +02:00
Shlomi Noach
896f560dce after timeout: reconnecting as new replica; skipping queries correctly 2016-05-23 11:12:59 +02:00
Shlomi Noach
5375aa4f69 - Removed use of master_pos_wait(). It was unneccessary in the first place and introduced new problems.
- Supporting `--allow-nullable-unique-key`
  - Tool will bail out if chosen key has nullable columns and the above is not provided
- Fixed `OriginalBinlogRowImage` comaprison (lower/upper case issue)
- Introduced reasonable streamer reconnect sleep time
2016-05-20 12:52:14 +02:00
Shlomi Noach
9b54d0208f - Handling gomysql.replication connection timeouts: reconnecting on last known position
- `printStatus()` takes ETA into account
- More info around `master_pos_wait()`
2016-05-19 15:11:36 +02:00
Shlomi Noach
ec34a5ef75 master_pos_wait is now OK to return NULL. We only care if it returns with -1 2016-05-18 15:08:47 +02:00
Shlomi Noach
9f56a84b57 Fixing single-row table migration
- `BuildUniqueKeyRangeEndPreparedQuery` supports `includeRangeStartValues` argument
- `applier` sends `this.migrationContext.GetIteration() == 0` as argument
2016-05-18 14:53:09 +02:00