* Cancel any row count queries before attempting to cut over
Closes#830. Switches from using `QueryRow` to `QueryRowContext`, and
stores a context.CancelFunc in the migration context, which is called to
halt any running row count query before beginning the cut over.
* Make it threadsafe
* Kill the count query on the database side as well
* Explicitly grab a connection to run the count, store its connection id
* When the query context is canceled, run a `KILL QUERY ?` on that connection id
* Rewrite these to use the threadsafe functions, stop exporting the cancel func
* Update logger
* Update logger
Co-authored-by: Tim Vaillancourt <timvaillancourt@github.com>
Co-authored-by: Tim Vaillancourt <tim@timvaillancourt.com>
Co-authored-by: dm-2 <45519614+dm-2@users.noreply.github.com>
* Add context/timeout to HTTP throttle check
* Dont run `.GetThrottleHTTPInterval()` on every loop
* Update help message
* Var rename
* 2022
* Add timeout flag
* Add unix/tcp server commands, use ParseInt() for string->int64
* Var rename
* Re-check http timeout on every loop iteration
* Remove stale comment
* Make throttle interval idempotent
* var rename
* Usage grammar
* Make http timeout idempotent too
* Parse time.Duration once
* Move timeout to NewThrottler
* Help update
* Set User-Agent header
* Re-add newline
Co-authored-by: dm-2 <45519614+dm-2@users.noreply.github.com>
* Check RowsAffected when applying DML events to get more accurate statistics
Addresses #600.
When applying a DML event, check the RowsAffected on the `Result`
struct. Since all DML event queries are point queries, this will only
ever be 0 or 1. The applier then takes this value and multiplies by
the `rowsDelta` of the event, resulting in a properly-signed, accurate
row delta to use in the statistics.
If an error occurs here, log it, but do not surface this as an
actual error .. simply assume the DML affected a row and move on. It
will be inaccurate, but this is already the case.
* Fix import
* update wording to warning log message
Co-authored-by: Tim Vaillancourt <timvaillancourt@github.com>
Co-authored-by: Tim Vaillancourt <timvaillancourt@github.com>
* Add a go.mod file
* run go mod vendor again
* Move to a well-supported ini file reader
* Remove GO111MODULE=off
* Use go 1.16
* Rename github.com/outbrain/golib -> github.com/openark/golib
* Remove *.go-e files
* Fix for `strconv.ParseInt: parsing "": invalid syntax` error
* Add test for '[osc]' section
Co-authored-by: Nate Wernimont <nate.wernimont@workiva.com>
When TLS hostname validation used for the MySQL connection, the
ServerName property needs to be set so that it knows which name to
validate on the certificate.
Without this option and with InsecureSkipVerify set to false, validation
will error here with a fatal error otherwise:
```
FATAL tls: either ServerName or InsecureSkipVerify must be specified in the tls.Config
```
* v1.1.0
* WIP: copying AUTO_INCREMENT value to ghost table
Initial commit: towards setting up a test suite
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
* greping for 'expect_table_structure' content
* Adding simple test for 'expect_table_structure' scenario
* adding tests for AUTO_INCREMENT value after row deletes. Should initially fail
* clear event beforehand
* parsing AUTO_INCREMENT from alter query, reading AUTO_INCREMENT from original table, applying AUTO_INCREMENT value onto ghost table if applicable and user has not specified AUTO_INCREMENT in alter statement
* support GetUint64
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
* minor update to test
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
* adding test for user defined AUTO_INCREMENT statement
* Generated column as part of UNIQUE (or PRIMARY) KEY
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
* skip analysis of generated column data type in unique key
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
* All MySQL DBs limited to max 3 concurrent/idle connections
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
* hooks: reporting GH_OST_ETA_SECONDS. ETA stored as part of migration context
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
* GH_OST_ETA_NANOSECONDS
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
* N/A denoted by negative value
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
* ETAUnknown constant
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
* Convering enum to varchar
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
* test: not null
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
* first attempt at setting enum-to-string right
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
* fix insert query
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
* store enum values, use when populating
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
* apply EnumValues to mapped column
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
* fix compilation error
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
* gofmt
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
* v1.1.0
* WIP: copying AUTO_INCREMENT value to ghost table
Initial commit: towards setting up a test suite
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
* greping for 'expect_table_structure' content
* Adding simple test for 'expect_table_structure' scenario
* adding tests for AUTO_INCREMENT value after row deletes. Should initially fail
* clear event beforehand
* parsing AUTO_INCREMENT from alter query, reading AUTO_INCREMENT from original table, applying AUTO_INCREMENT value onto ghost table if applicable and user has not specified AUTO_INCREMENT in alter statement
* support GetUint64
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
* minor update to test
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
* adding test for user defined AUTO_INCREMENT statement
* Generated column as part of UNIQUE (or PRIMARY) KEY
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
* skip analysis of generated column data type in unique key
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
* All MySQL DBs limited to max 3 concurrent/idle connections
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
* hooks: reporting GH_OST_ETA_SECONDS. ETA stored as part of migration context
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
* GH_OST_ETA_NANOSECONDS
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
* N/A denoted by negative value
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
* ETAUnknown constant
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
* v1.1.0
* WIP: copying AUTO_INCREMENT value to ghost table
Initial commit: towards setting up a test suite
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
* greping for 'expect_table_structure' content
* Adding simple test for 'expect_table_structure' scenario
* adding tests for AUTO_INCREMENT value after row deletes. Should initially fail
* clear event beforehand
* parsing AUTO_INCREMENT from alter query, reading AUTO_INCREMENT from original table, applying AUTO_INCREMENT value onto ghost table if applicable and user has not specified AUTO_INCREMENT in alter statement
* support GetUint64
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
* minor update to test
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
* adding test for user defined AUTO_INCREMENT statement
* Generated column as part of UNIQUE (or PRIMARY) KEY
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
* skip analysis of generated column data type in unique key
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
* All MySQL DBs limited to max 3 concurrent/idle connections
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
* v1.1.0
* WIP: copying AUTO_INCREMENT value to ghost table
Initial commit: towards setting up a test suite
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
* greping for 'expect_table_structure' content
* Adding simple test for 'expect_table_structure' scenario
* adding tests for AUTO_INCREMENT value after row deletes. Should initially fail
* clear event beforehand
* parsing AUTO_INCREMENT from alter query, reading AUTO_INCREMENT from original table, applying AUTO_INCREMENT value onto ghost table if applicable and user has not specified AUTO_INCREMENT in alter statement
* support GetUint64
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
* minor update to test
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
* adding test for user defined AUTO_INCREMENT statement
* Generated column as part of UNIQUE (or PRIMARY) KEY
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
* skip analysis of generated column data type in unique key
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
* v1.1.0
* WIP: copying AUTO_INCREMENT value to ghost table
Initial commit: towards setting up a test suite
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
* greping for 'expect_table_structure' content
* Adding simple test for 'expect_table_structure' scenario
* adding tests for AUTO_INCREMENT value after row deletes. Should initially fail
* clear event beforehand
* parsing AUTO_INCREMENT from alter query, reading AUTO_INCREMENT from original table, applying AUTO_INCREMENT value onto ghost table if applicable and user has not specified AUTO_INCREMENT in alter statement
* support GetUint64
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
* minor update to test
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
* adding test for user defined AUTO_INCREMENT statement
Co-authored-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
Closes#822.
In https://github.com/go-sql-driver/mysql/issues/1075, @acharis notes
that the way the go-sql driver is written, query timeout errors don't
get set in `rows.Err()` until _after_ a call to `rows.Next()` is made.
Because this kind of error means there will be no rows in the result
set, the `for rows.Next()` will never enter the for loop, so we must
check the value of `rows.Err()` after the loop, and surface the error up
appropriately.
