Commit Graph

33 Commits

Author SHA1 Message Date
Tim Vaillancourt
ffe54f48ed
Enable more golang-ci linters (#1149) 2022-07-18 18:37:18 +02:00
Tim Vaillancourt
500fbefdb9 golang-ci: fix staticcheck linter warnings 2022-07-07 03:12:44 +02:00
Tim Vaillancourt
ed46138c06
Add golangci-lint CI action, fix gosimple, govet + unused lint errors (#1127)
* Add `golangci-lint`, fix `gosimple`, `govet` and `unused` linter complaints

* Go 1.16

* Update copyright dates
2022-05-31 21:23:39 +02:00
Shlomi Noach
9bc508f068
Enum to varchar (#963)
* v1.1.0

* WIP: copying AUTO_INCREMENT value to ghost table
Initial commit: towards setting up a test suite

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* greping for 'expect_table_structure' content

* Adding simple test for 'expect_table_structure' scenario

* adding tests for AUTO_INCREMENT value after row deletes. Should initially fail

* clear event beforehand

* parsing AUTO_INCREMENT from alter query, reading AUTO_INCREMENT from original table, applying AUTO_INCREMENT value onto ghost table if applicable and user has not specified AUTO_INCREMENT in alter statement

* support GetUint64

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* minor update to test

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* adding test for user defined AUTO_INCREMENT statement

* Generated column as part of UNIQUE (or PRIMARY) KEY

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* skip analysis of generated column data type in unique key

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* All MySQL DBs limited to max 3 concurrent/idle connections

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* hooks: reporting GH_OST_ETA_SECONDS. ETA stored as part of migration context

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* GH_OST_ETA_NANOSECONDS

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* N/A denoted by negative value

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* ETAUnknown constant

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* Convering enum to varchar

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* test: not null

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* first attempt at setting enum-to-string right

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* fix insert query

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* store enum values, use when populating

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* apply EnumValues to mapped column

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* fix compilation error

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* gofmt

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
2021-06-10 17:17:49 +02:00
thsun
40bc5aedf2 enhance processing efficiency 2021-02-26 14:23:24 +08:00
Yaser Azfar
2eb141010b adds error checking for an err variable that was left unchecked 2019-12-27 16:54:43 +13:00
MOON_CLJ
21d455013e minor changes and typo correction 2018-05-23 13:13:25 +08:00
Atsushi Ishibashi
99fcbc35f3 Fix custom type assignment 2018-05-04 01:45:14 +09:00
Shlomi Noach
c9f2da82f0 Merge branch 'master' into json-57-test 2017-08-27 08:01:02 +03:00
Shlomi Noach
2174588ebf Supporting JSON types 2017-08-22 11:51:20 +03:00
Shlomi Noach
79e4f1cdbe detect range end based on OFFSET 2017-08-21 08:12:41 +03:00
Shlomi Noach
a07a6f8cf5 fixing mediumint unsigned problem 2017-01-10 09:57:42 +02:00
Shlomi Noach
25166e33c7 solving the enum-as-part-of-pk bug 2016-10-19 15:22:29 +02:00
Shlomi Noach
6fbf337051 builder is smarter and can use convert_tz where it notices DATETIME->TIMESTAMP conversion 2016-10-13 13:11:18 +02:00
Shlomi Noach
51f0c0d957 fix arg conversion in BuildDMLUpdateQuery 2016-09-27 13:08:31 +02:00
Paulo Bittencourt
2f80c9d424 Used the source column definition to fetch and convert insert values
When processing binlog insert statements, we want to use `sharedColumns`
to decide which values to fetch and convert from the insert DML event.

We only want to  `mappedShareColumns` to define the column names in the
`replace into ...` statement.
2016-09-14 11:28:30 -04:00
Shlomi Noach
791d963ea0 Character set recognition and manipulation
- Identifying textual characters sets; converting into specific type when applying dml events
- Refactored `ColumnsList`: introducing `Column` type
- Refactored `unsigned` handling, as part of `Column`
- `Column` type supports `convertArg()`: converting value of argument according to column data type
- DB URI attempts `utf8mb4,utf8,latin1` charsets in that order (first one to be recognized wins)
- Local tests filter by pattern
- Local tests append table schema on failure
- Local tests do not have postpone flag file
- Added character set local tests: `utf8`, `utf8mb4`, `latin1`
2016-09-07 14:24:11 +02:00
Shlomi Noach
b63cc3e75e fix INSERT DML handling on renamed column 2016-08-22 16:00:15 +02:00
Shlomi Noach
1376f0af23 fixed UPDATE dml on renamed column 2016-08-22 08:49:27 +02:00
Shlomi Noach
74593ec010 DML write wrapped in transaction
- solving the golang problem: 'sql: converting Exec argument #2's type: uint64 values with high bit set are not supported'
2016-08-18 13:31:53 +02:00
Shlomi Noach
16d76aa299 builder fixes arg type from signed to unsigned based on UnsingedFlags 2016-08-17 06:52:23 +02:00
Shlomi Noach
836d0fe119 Supporting column rename
- Parsing `alter` statement to catch `change old_name new_name ...` statements
- Auto deducing renamed columns
- When suspecting renamed columns, requesting explicit `--approve-renamed-columns` or `--skip-renamed-columns`
- updated tests
2016-06-17 08:03:18 +02:00
Shlomi Noach
9f56a84b57 Fixing single-row table migration
- `BuildUniqueKeyRangeEndPreparedQuery` supports `includeRangeStartValues` argument
- `applier` sends `this.migrationContext.GetIteration() == 0` as argument
2016-05-18 14:53:09 +02:00
Shlomi Noach
9d055dbda7 renaming to gh-ost 2016-05-16 11:09:17 +02:00
Shlomi Noach
a4ee80df13 - Building and applying queries from binlog event data!
- `INSERT`, `DELETE`, `UPDATE` statements
- support for `--noop`
- initial support for `--test-on-replica`. Verifying against `--allow-on-master`
- Changelog events no longer read from binlog stream, because reading it may be throttled, and we have to be able to keep reading the heartbeat and state events.
  They are now being read directly from table, mapping already-seen-events to avoid confusion
  Changlelog listener pools table in 2*frequency of heartbeat injection
2016-04-14 13:37:56 +02:00
Shlomi Noach
04525887f3 - Throttling-check is now an async routine running once per second
- Throttling variables protected by mutex
- Added `--throttle-additional-flag-file`: `operation pauses when this file exists; hint: keep default, use for throttling multiple gh-osc operations`
- ColumnList is not a `struct` which contains ordinal mapping
- More implicit write changelog + audit changelog
- builder now builds `DELETE` and `INSERT` queries from data it will eventually get from DML event
- Sanity check for binlog_row_image
- Restarting replication to be sure binlog settings apply
- Prepare for accepting `SIGHUP` (reloading configuration)
2016-04-11 17:27:16 +02:00
Shlomi Noach
75f68c0752 - row copy and row events are now handled by a single routine which prioritizes events over rowcopy
- Supporting `--throttle-file-flag`
- Printing status
- Supporting transactional table syntax
- code cleanup; refactoring
- proper use of atomic where required
- iterations are in changelog (erm... maybe too much)
- `LOCK TABLES`, `UNLOCK TABLES` working
2016-04-08 10:34:44 +02:00
Shlomi Noach
d8fefb3d6f exploded args on range query building; iteration works 2016-04-05 19:50:49 +02:00
Shlomi Noach
3583ab5dc5 beginning support for ranges and iteration. Still WIP 2016-04-05 09:14:22 +02:00
Shlomi Noach
ea0906f4e5 reading table (range) min/max values, right now according to hardcoded unique key 2016-04-04 18:19:46 +02:00
Shlomi Noach
b461c320cc added BuildUniqueKeyRangeEndPreparedQuery() 2016-04-01 16:05:17 +02:00
Shlomi Noach
41c3d15963 building prepared queries (with positional params) 2016-04-01 13:55:51 +02:00
Shlomi Noach
39ebc75c43 initial work on sql query building 2016-04-01 13:36:56 +02:00