Commit Graph

68 Commits

Author SHA1 Message Date
wangzihuacool
113094bbd2 Convert character to bytes and insert into table using latin1 2022-08-03 11:43:18 +08:00
Tim Vaillancourt
ffe54f48ed
Enable more golang-ci linters (#1149) 2022-07-18 18:37:18 +02:00
Tim Vaillancourt
500fbefdb9 golang-ci: fix staticcheck linter warnings 2022-07-07 03:12:44 +02:00
Tim Vaillancourt
ed46138c06
Add golangci-lint CI action, fix gosimple, govet + unused lint errors (#1127)
* Add `golangci-lint`, fix `gosimple`, `govet` and `unused` linter complaints

* Go 1.16

* Update copyright dates
2022-05-31 21:23:39 +02:00
Tim Vaillancourt
47d49c6b92
Add go mod (#935)
* Add a go.mod file

* run go mod vendor again

* Move to a well-supported ini file reader

* Remove GO111MODULE=off

* Use go 1.16

* Rename github.com/outbrain/golib -> github.com/openark/golib

* Remove *.go-e files

* Fix for `strconv.ParseInt: parsing "": invalid syntax` error

* Add test for '[osc]' section

Co-authored-by: Nate Wernimont <nate.wernimont@workiva.com>
2021-06-24 20:19:37 +02:00
Shlomi Noach
9bc508f068
Enum to varchar (#963)
* v1.1.0

* WIP: copying AUTO_INCREMENT value to ghost table
Initial commit: towards setting up a test suite

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* greping for 'expect_table_structure' content

* Adding simple test for 'expect_table_structure' scenario

* adding tests for AUTO_INCREMENT value after row deletes. Should initially fail

* clear event beforehand

* parsing AUTO_INCREMENT from alter query, reading AUTO_INCREMENT from original table, applying AUTO_INCREMENT value onto ghost table if applicable and user has not specified AUTO_INCREMENT in alter statement

* support GetUint64

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* minor update to test

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* adding test for user defined AUTO_INCREMENT statement

* Generated column as part of UNIQUE (or PRIMARY) KEY

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* skip analysis of generated column data type in unique key

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* All MySQL DBs limited to max 3 concurrent/idle connections

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* hooks: reporting GH_OST_ETA_SECONDS. ETA stored as part of migration context

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* GH_OST_ETA_NANOSECONDS

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* N/A denoted by negative value

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* ETAUnknown constant

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* Convering enum to varchar

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* test: not null

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* first attempt at setting enum-to-string right

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* fix insert query

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* store enum values, use when populating

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* apply EnumValues to mapped column

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* fix compilation error

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* gofmt

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
2021-06-10 17:17:49 +02:00
Tim Vaillancourt
c71dbf9ef3
Copy auto increment (#967)
* v1.1.0

* WIP: copying AUTO_INCREMENT value to ghost table
Initial commit: towards setting up a test suite

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* greping for 'expect_table_structure' content

* Adding simple test for 'expect_table_structure' scenario

* adding tests for AUTO_INCREMENT value after row deletes. Should initially fail

* clear event beforehand

* parsing AUTO_INCREMENT from alter query, reading AUTO_INCREMENT from original table, applying AUTO_INCREMENT value onto ghost table if applicable and user has not specified AUTO_INCREMENT in alter statement

* support GetUint64

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* minor update to test

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* adding test for user defined AUTO_INCREMENT statement

Co-authored-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
2021-05-14 15:32:56 +02:00
thsun
40bc5aedf2 enhance processing efficiency 2021-02-26 14:23:24 +08:00
thsun
960d801276 canonical code 2021-02-24 17:18:21 +08:00
thsun
2b5d5e0a11 gofmt 2021-01-14 14:51:14 +08:00
thsun
d0f0b95cf8 rm main 2021-01-13 17:21:21 +08:00
Tim Vaillancourt
10724f08c5 Merge branch 'parse-alter-statement' of https://github.com/openark/gh-ost into parse-alter-statement 2020-08-19 21:39:36 +02:00
Shlomi Noach
ae4dd1867a extra unit test checks 2020-07-29 15:06:13 +03:00
Shlomi Noach
731df3cd15 comments 2020-07-23 14:04:14 +03:00
Shlomi Noach
c9249f2b71 Updating and using AlterTableOptions 2020-07-23 11:38:05 +03:00
Shlomi Noach
6c7b4736e1 Support a complete ALTER TABLE statement in --alter 2020-07-22 12:33:02 +03:00
Yaser Azfar
2eb141010b adds error checking for an err variable that was left unchecked 2019-12-27 16:54:43 +13:00
MOON_CLJ
21d455013e minor changes and typo correction 2018-05-23 13:13:25 +08:00
Shlomi Noach
24ed4ce5e6
Merge branch 'master' into reject-rename-table 2018-05-07 08:04:47 +03:00
Shlomi Noach
fb00a13871 Rejecting RENAME TO|AS 2018-05-06 11:19:03 +03:00
Atsushi Ishibashi
99fcbc35f3 Fix custom type assignment 2018-05-04 01:45:14 +09:00
ceshihao
35becabf3c support gbk encoding 2018-01-16 23:43:09 +08:00
Josh Soref
e99333e576 spelling: conversion 2017-11-08 00:45:31 +00:00
Shlomi Noach
3437cf44c4 Validating shared key column types 2017-09-03 09:57:26 +03:00
Shlomi Noach
c9f2da82f0 Merge branch 'master' into json-57-test 2017-08-27 08:01:02 +03:00
Shlomi Noach
2174588ebf Supporting JSON types 2017-08-22 11:51:20 +03:00
Shlomi Noach
79e4f1cdbe detect range end based on OFFSET 2017-08-21 08:12:41 +03:00
Shlomi Noach
8a0f1413eb dropped columns are not 'shared' and no data copy attempted for such columns 2017-04-23 08:38:35 +03:00
Shlomi Noach
85c498511e parser recognizes DROP COLUMN tokens 2017-04-23 08:23:56 +03:00
Shlomi Noach
a07a6f8cf5 fixing mediumint unsigned problem 2017-01-10 09:57:42 +02:00
Shlomi Noach
3fd85ee8b1 test logging cleanup 2016-12-11 02:22:56 +01:00
Shlomi Noach
0a707688e0 added decimal test 2016-11-29 15:47:39 +01:00
Shlomi Noach
e7cf488818 fixed parsing of quotes and of detecting rename statements 2016-11-29 15:44:38 +01:00
Shlomi Noach
25166e33c7 solving the enum-as-part-of-pk bug 2016-10-19 15:22:29 +02:00
Shlomi Noach
9ac468c97e failing on DATETIME-to-TIMESTAMP conversion within chosen key 2016-10-14 09:33:53 +02:00
Shlomi Noach
661b7ff0c1 added test to sql/types 2016-10-13 13:13:00 +02:00
Shlomi Noach
882959ba83 column is type-aware; is timezone-conversion aware
also refactored some functions
2016-10-13 13:12:33 +02:00
Shlomi Noach
a637ed1b50 adapted builder tests 2016-10-13 13:11:35 +02:00
Shlomi Noach
6fbf337051 builder is smarter and can use convert_tz where it notices DATETIME->TIMESTAMP conversion 2016-10-13 13:11:18 +02:00
Shlomi Noach
51f0c0d957 fix arg conversion in BuildDMLUpdateQuery 2016-09-27 13:08:31 +02:00
Paulo Bittencourt
2f80c9d424 Used the source column definition to fetch and convert insert values
When processing binlog insert statements, we want to use `sharedColumns`
to decide which values to fetch and convert from the insert DML event.

We only want to  `mappedShareColumns` to define the column names in the
`replace into ...` statement.
2016-09-14 11:28:30 -04:00
Shlomi Noach
f4693274fb Extracted and generalized encoding logic to encoding.go 2016-09-08 09:37:22 +02:00
Shlomi Noach
791d963ea0 Character set recognition and manipulation
- Identifying textual characters sets; converting into specific type when applying dml events
- Refactored `ColumnsList`: introducing `Column` type
- Refactored `unsigned` handling, as part of `Column`
- `Column` type supports `convertArg()`: converting value of argument according to column data type
- DB URI attempts `utf8mb4,utf8,latin1` charsets in that order (first one to be recognized wins)
- Local tests filter by pattern
- Local tests append table schema on failure
- Local tests do not have postpone flag file
- Added character set local tests: `utf8`, `utf8mb4`, `latin1`
2016-09-07 14:24:11 +02:00
Shlomi Noach
b63cc3e75e fix INSERT DML handling on renamed column 2016-08-22 16:00:15 +02:00
Shlomi Noach
1376f0af23 fixed UPDATE dml on renamed column 2016-08-22 08:49:27 +02:00
Shlomi Noach
74593ec010 DML write wrapped in transaction
- solving the golang problem: 'sql: converting Exec argument #2's type: uint64 values with high bit set are not supported'
2016-08-18 13:31:53 +02:00
Shlomi Noach
16d76aa299 builder fixes arg type from signed to unsigned based on UnsingedFlags 2016-08-17 06:52:23 +02:00
Shlomi Noach
7a0e2cfe3f adding UnsignedFlags to ColumnsList 2016-08-17 06:49:20 +02:00
Daniël van Eeden
7786356a45 Fix typo: s/auto_incrmenet/auto_increment/ 2016-08-05 17:46:38 +02:00
Shlomi Noach
23cb8ea7e9 Throttling & critical load
- Added `--throttle-query` param (when returns > 0, throttling applies)
- Added `--critical-load`, similar to `--max-load` but implies panic and quit
- Recoded *-load as `LoadMap`
- More info on *-load throttle/panic
- `printStatus()` now gets printing heuristic. Always shows up on interactive `"status"`
- Fixed `change column` (aka rename) handling with quotes
- Removed legacy `mysqlbinlog` parser code
- Added tests
2016-06-18 21:12:07 +02:00