* Cancel any row count queries before attempting to cut over
Closes#830. Switches from using `QueryRow` to `QueryRowContext`, and
stores a context.CancelFunc in the migration context, which is called to
halt any running row count query before beginning the cut over.
* Make it threadsafe
* Kill the count query on the database side as well
* Explicitly grab a connection to run the count, store its connection id
* When the query context is canceled, run a `KILL QUERY ?` on that connection id
* Rewrite these to use the threadsafe functions, stop exporting the cancel func
* Update logger
* Update logger
Co-authored-by: Tim Vaillancourt <timvaillancourt@github.com>
Co-authored-by: Tim Vaillancourt <tim@timvaillancourt.com>
Co-authored-by: dm-2 <45519614+dm-2@users.noreply.github.com>
* Add a go.mod file
* run go mod vendor again
* Move to a well-supported ini file reader
* Remove GO111MODULE=off
* Use go 1.16
* Rename github.com/outbrain/golib -> github.com/openark/golib
* Remove *.go-e files
* Fix for `strconv.ParseInt: parsing "": invalid syntax` error
* Add test for '[osc]' section
Co-authored-by: Nate Wernimont <nate.wernimont@workiva.com>
* v1.1.0
* WIP: copying AUTO_INCREMENT value to ghost table
Initial commit: towards setting up a test suite
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
* greping for 'expect_table_structure' content
* Adding simple test for 'expect_table_structure' scenario
* adding tests for AUTO_INCREMENT value after row deletes. Should initially fail
* clear event beforehand
* parsing AUTO_INCREMENT from alter query, reading AUTO_INCREMENT from original table, applying AUTO_INCREMENT value onto ghost table if applicable and user has not specified AUTO_INCREMENT in alter statement
* support GetUint64
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
* minor update to test
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
* adding test for user defined AUTO_INCREMENT statement
* Generated column as part of UNIQUE (or PRIMARY) KEY
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
* skip analysis of generated column data type in unique key
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
* All MySQL DBs limited to max 3 concurrent/idle connections
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
* hooks: reporting GH_OST_ETA_SECONDS. ETA stored as part of migration context
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
* GH_OST_ETA_NANOSECONDS
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
* N/A denoted by negative value
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
* ETAUnknown constant
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
* Convering enum to varchar
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
* test: not null
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
* first attempt at setting enum-to-string right
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
* fix insert query
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
* store enum values, use when populating
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
* apply EnumValues to mapped column
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
* fix compilation error
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
* gofmt
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
* v1.1.0
* WIP: copying AUTO_INCREMENT value to ghost table
Initial commit: towards setting up a test suite
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
* greping for 'expect_table_structure' content
* Adding simple test for 'expect_table_structure' scenario
* adding tests for AUTO_INCREMENT value after row deletes. Should initially fail
* clear event beforehand
* parsing AUTO_INCREMENT from alter query, reading AUTO_INCREMENT from original table, applying AUTO_INCREMENT value onto ghost table if applicable and user has not specified AUTO_INCREMENT in alter statement
* support GetUint64
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
* minor update to test
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
* adding test for user defined AUTO_INCREMENT statement
* Generated column as part of UNIQUE (or PRIMARY) KEY
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
* skip analysis of generated column data type in unique key
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
* v1.1.0
* WIP: copying AUTO_INCREMENT value to ghost table
Initial commit: towards setting up a test suite
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
* greping for 'expect_table_structure' content
* Adding simple test for 'expect_table_structure' scenario
* adding tests for AUTO_INCREMENT value after row deletes. Should initially fail
* clear event beforehand
* parsing AUTO_INCREMENT from alter query, reading AUTO_INCREMENT from original table, applying AUTO_INCREMENT value onto ghost table if applicable and user has not specified AUTO_INCREMENT in alter statement
* support GetUint64
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
* minor update to test
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
* adding test for user defined AUTO_INCREMENT statement
Co-authored-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
The TABLE_SCHEMA and TABLE_NAME are already guaranteed to have the same value in COLUMNS and UNIQUES because of the WHEREs in each query. Dropping it from the ON clause makes it complete much faster.
On (at least) MySQL 5.6 installs with thousands of tables, this query completes in a few seconds without the additional join conditions, but takes more than a minute with it.
There is no need to call `applyColumnTypes` more than once for the same
`databaseName` and `tableName`, it is just move the additional
`columnList` to the first call.
The given `columnLists` may not contain all columns available in the
given table. This patch prevents the code to fail. Before this patch the
code was panicking whenever it was trying to set a value into a `nil`
object, e.g. `columnList.GetColumn('non-existant').Type = SomeType`.
For some reason the application was not completely failing but as a
side-effect any column after the non-existant column would never get its
Type properly set.
Both Percona and Maria allow MySQL to be configured to listen on an extra port when their thread pool is enable.
* https://www.percona.com/doc/percona-server/5.7/performance/threadpool.html
* https://mariadb.com/kb/en/the-mariadb-library/thread-pool-in-mariadb-51-53/
This is valuable because if the table has a lot of traffic (read or write load), gh-ost can end up starving the thread pool as incomming connections are immediately blocked.
By using gh-ost on the extra port, MySQL locking will still behave the same, but MySQL will keep a dedicated thread for each gh-ost connection.
When doing this, it's important to inspect the extra-max-connections variable. Both Percona and Maria default to 1, so gh-ost may easily exceed with its threads.
An example local run using this
```
$ mysql -S /tmp/mysql_sandbox20393.sock -e "select @@global.port, @@global.extra_port"
+---------------+---------------------+
| @@global.port | @@global.extra_port |
+---------------+---------------------+
| 20393 | 30393 |
+---------------+---------------------+
./bin/gh-ost \
--initially-drop-ghost-table \
--initially-drop-old-table \
--assume-rbr \
--port="20395" \
--assume-master-host="127.0.0.1:30393" \
--max-load=Threads_running=25 \
--critical-load=Threads_running=1000 \
--chunk-size=1000 \
--max-lag-millis=1500 \
--user="gh-ost" \
--password="gh-ost" \
--database="test" \
--table="mytable" \
--verbose \
--alter="ADD mynewcol decimal(11,2) DEFAULT 0.0 NOT NULL" \
--exact-rowcount \
--concurrent-rowcount \
--default-retries=120 \
--panic-flag-file=/tmp/ghost.panic.flag \
--postpone-cut-over-flag-file=/tmp/ghost.postpone.flag \
--execute
```
