Commit Graph

57 Commits

Author SHA1 Message Date
Tim Vaillancourt
10724f08c5 Merge branch 'parse-alter-statement' of https://github.com/openark/gh-ost into parse-alter-statement 2020-08-19 21:39:36 +02:00
Shlomi Noach
ae4dd1867a extra unit test checks 2020-07-29 15:06:13 +03:00
Shlomi Noach
731df3cd15 comments 2020-07-23 14:04:14 +03:00
Shlomi Noach
c9249f2b71 Updating and using AlterTableOptions 2020-07-23 11:38:05 +03:00
Shlomi Noach
6c7b4736e1 Support a complete ALTER TABLE statement in --alter 2020-07-22 12:33:02 +03:00
Yaser Azfar
2eb141010b adds error checking for an err variable that was left unchecked 2019-12-27 16:54:43 +13:00
MOON_CLJ
21d455013e minor changes and typo correction 2018-05-23 13:13:25 +08:00
Shlomi Noach
24ed4ce5e6
Merge branch 'master' into reject-rename-table 2018-05-07 08:04:47 +03:00
Shlomi Noach
fb00a13871 Rejecting RENAME TO|AS 2018-05-06 11:19:03 +03:00
Atsushi Ishibashi
99fcbc35f3 Fix custom type assignment 2018-05-04 01:45:14 +09:00
ceshihao
35becabf3c support gbk encoding 2018-01-16 23:43:09 +08:00
Josh Soref
e99333e576 spelling: conversion 2017-11-08 00:45:31 +00:00
Shlomi Noach
3437cf44c4 Validating shared key column types 2017-09-03 09:57:26 +03:00
Shlomi Noach
c9f2da82f0 Merge branch 'master' into json-57-test 2017-08-27 08:01:02 +03:00
Shlomi Noach
2174588ebf Supporting JSON types 2017-08-22 11:51:20 +03:00
Shlomi Noach
79e4f1cdbe detect range end based on OFFSET 2017-08-21 08:12:41 +03:00
Shlomi Noach
8a0f1413eb dropped columns are not 'shared' and no data copy attempted for such columns 2017-04-23 08:38:35 +03:00
Shlomi Noach
85c498511e parser recognizes DROP COLUMN tokens 2017-04-23 08:23:56 +03:00
Shlomi Noach
a07a6f8cf5 fixing mediumint unsigned problem 2017-01-10 09:57:42 +02:00
Shlomi Noach
3fd85ee8b1 test logging cleanup 2016-12-11 02:22:56 +01:00
Shlomi Noach
0a707688e0 added decimal test 2016-11-29 15:47:39 +01:00
Shlomi Noach
e7cf488818 fixed parsing of quotes and of detecting rename statements 2016-11-29 15:44:38 +01:00
Shlomi Noach
25166e33c7 solving the enum-as-part-of-pk bug 2016-10-19 15:22:29 +02:00
Shlomi Noach
9ac468c97e failing on DATETIME-to-TIMESTAMP conversion within chosen key 2016-10-14 09:33:53 +02:00
Shlomi Noach
661b7ff0c1 added test to sql/types 2016-10-13 13:13:00 +02:00
Shlomi Noach
882959ba83 column is type-aware; is timezone-conversion aware
also refactored some functions
2016-10-13 13:12:33 +02:00
Shlomi Noach
a637ed1b50 adapted builder tests 2016-10-13 13:11:35 +02:00
Shlomi Noach
6fbf337051 builder is smarter and can use convert_tz where it notices DATETIME->TIMESTAMP conversion 2016-10-13 13:11:18 +02:00
Shlomi Noach
51f0c0d957 fix arg conversion in BuildDMLUpdateQuery 2016-09-27 13:08:31 +02:00
Paulo Bittencourt
2f80c9d424 Used the source column definition to fetch and convert insert values
When processing binlog insert statements, we want to use `sharedColumns`
to decide which values to fetch and convert from the insert DML event.

We only want to  `mappedShareColumns` to define the column names in the
`replace into ...` statement.
2016-09-14 11:28:30 -04:00
Shlomi Noach
f4693274fb Extracted and generalized encoding logic to encoding.go 2016-09-08 09:37:22 +02:00
Shlomi Noach
791d963ea0 Character set recognition and manipulation
- Identifying textual characters sets; converting into specific type when applying dml events
- Refactored `ColumnsList`: introducing `Column` type
- Refactored `unsigned` handling, as part of `Column`
- `Column` type supports `convertArg()`: converting value of argument according to column data type
- DB URI attempts `utf8mb4,utf8,latin1` charsets in that order (first one to be recognized wins)
- Local tests filter by pattern
- Local tests append table schema on failure
- Local tests do not have postpone flag file
- Added character set local tests: `utf8`, `utf8mb4`, `latin1`
2016-09-07 14:24:11 +02:00
Shlomi Noach
b63cc3e75e fix INSERT DML handling on renamed column 2016-08-22 16:00:15 +02:00
Shlomi Noach
1376f0af23 fixed UPDATE dml on renamed column 2016-08-22 08:49:27 +02:00
Shlomi Noach
74593ec010 DML write wrapped in transaction
- solving the golang problem: 'sql: converting Exec argument #2's type: uint64 values with high bit set are not supported'
2016-08-18 13:31:53 +02:00
Shlomi Noach
16d76aa299 builder fixes arg type from signed to unsigned based on UnsingedFlags 2016-08-17 06:52:23 +02:00
Shlomi Noach
7a0e2cfe3f adding UnsignedFlags to ColumnsList 2016-08-17 06:49:20 +02:00
Daniël van Eeden
7786356a45 Fix typo: s/auto_incrmenet/auto_increment/ 2016-08-05 17:46:38 +02:00
Shlomi Noach
23cb8ea7e9 Throttling & critical load
- Added `--throttle-query` param (when returns > 0, throttling applies)
- Added `--critical-load`, similar to `--max-load` but implies panic and quit
- Recoded *-load as `LoadMap`
- More info on *-load throttle/panic
- `printStatus()` now gets printing heuristic. Always shows up on interactive `"status"`
- Fixed `change column` (aka rename) handling with quotes
- Removed legacy `mysqlbinlog` parser code
- Added tests
2016-06-18 21:12:07 +02:00
Shlomi Noach
836d0fe119 Supporting column rename
- Parsing `alter` statement to catch `change old_name new_name ...` statements
- Auto deducing renamed columns
- When suspecting renamed columns, requesting explicit `--approve-renamed-columns` or `--skip-renamed-columns`
- updated tests
2016-06-17 08:03:18 +02:00
Shlomi Noach
9f56a84b57 Fixing single-row table migration
- `BuildUniqueKeyRangeEndPreparedQuery` supports `includeRangeStartValues` argument
- `applier` sends `this.migrationContext.GetIteration() == 0` as argument
2016-05-18 14:53:09 +02:00
Shlomi Noach
9d055dbda7 renaming to gh-ost 2016-05-16 11:09:17 +02:00
Shlomi Noach
eeffa701d6 - Added ok-to-drop-table flag
- Added `switch-to-rbr` flag; applying binlog format change if needed
- Using dedicated db instance for locking & renaming on applier (must be used from within same connection)
- Heartbeat now uses `time.RFC3339Nano`
- Swap tables works! Caveat: short table outage
- `--test-on-replica` works!
- retries: using `panicAbort`: from any goroutine, regardless of context, it is possible to terminate the operation
- Reintroduced changelog events listener on streamer. This is the correct implementation.
2016-04-18 10:57:18 -07:00
Shlomi Noach
a4ee80df13 - Building and applying queries from binlog event data!
- `INSERT`, `DELETE`, `UPDATE` statements
- support for `--noop`
- initial support for `--test-on-replica`. Verifying against `--allow-on-master`
- Changelog events no longer read from binlog stream, because reading it may be throttled, and we have to be able to keep reading the heartbeat and state events.
  They are now being read directly from table, mapping already-seen-events to avoid confusion
  Changlelog listener pools table in 2*frequency of heartbeat injection
2016-04-14 13:37:56 +02:00
Shlomi Noach
0d25d11b40 added types_test 2016-04-11 19:06:47 +02:00
Shlomi Noach
04525887f3 - Throttling-check is now an async routine running once per second
- Throttling variables protected by mutex
- Added `--throttle-additional-flag-file`: `operation pauses when this file exists; hint: keep default, use for throttling multiple gh-osc operations`
- ColumnList is not a `struct` which contains ordinal mapping
- More implicit write changelog + audit changelog
- builder now builds `DELETE` and `INSERT` queries from data it will eventually get from DML event
- Sanity check for binlog_row_image
- Restarting replication to be sure binlog settings apply
- Prepare for accepting `SIGHUP` (reloading configuration)
2016-04-11 17:27:16 +02:00
Shlomi Noach
a1a34b8150 ongoing development:
- accepts --max-load
- accepts multiple conditions in --max-load
- throttle includes reason
- chunk-size sanity check
- change log state writes both in appending (history) mode and in replacing (current) mode
- more atomic checks
- inspecting ghost table columns, unique key
- comparing unique keys between tables; sanity
- intersecting columns between tables
- prettify status
- refactored throttle() and retries()
2016-04-08 14:35:06 +02:00
Shlomi Noach
75f68c0752 - row copy and row events are now handled by a single routine which prioritizes events over rowcopy
- Supporting `--throttle-file-flag`
- Printing status
- Supporting transactional table syntax
- code cleanup; refactoring
- proper use of atomic where required
- iterations are in changelog (erm... maybe too much)
- `LOCK TABLES`, `UNLOCK TABLES` working
2016-04-08 10:34:44 +02:00
Shlomi Noach
0e7b23e6fe - Creating an populating Changelog table
- Using heartbeat
- Throttling works based on heartbeat
- Refactored binlog_reader stuff. Now streaming events (into golang channel, which makes for nice buffering and throttling)
- Binlog table listeners work
- More Migrator logic; existing logic for waiting on `state` events (e.g. `TablesCreatedState`)
2016-04-07 15:57:12 +02:00
Shlomi Noach
4dd5a93ed7 - More work on go-mysql reader: now properly notes binlog coordinates as it makes progress
- Properly creating multiple BinlogEntry-ies when reading multi statement binlog event (Still todo: transactions)
2016-04-06 18:44:54 +02:00