Shlomi Noach
1671530300
Merge branch 'master' into hints-owner-token
2019-01-28 08:28:58 +02:00
Shlomi Noach
3161cd5823
added: --hooks-hint-owner, --hooks-hint-token
2019-01-24 13:06:20 +02:00
Shlomi Noach
93b88e262a
Auto-merged master into gh-ost/patch-1 on deployment
2019-01-23 08:13:55 +02:00
twotwotwo
afa108f8fa
inspect: remove redundant join conditions
...
The TABLE_SCHEMA and TABLE_NAME are already guaranteed to have the same value in COLUMNS and UNIQUES because of the WHEREs in each query. Dropping it from the ON clause makes it complete much faster.
On (at least) MySQL 5.6 installs with thousands of tables, this query completes in a few seconds without the additional join conditions, but takes more than a minute with it.
2019-01-22 12:26:55 -08:00
Shlomi Noach
809908cb1d
Merge branch 'master' into ipv6
2019-01-20 09:02:34 +02:00
Shlomi Noach
5f587c3262
Adding --force-named-panic option
2019-01-14 13:27:44 +02:00
Shlomi Noach
d32304520c
Merge branch 'master' into minor_changes
2019-01-14 10:45:27 +02:00
Shlomi Noach
7c17cee010
reverting some changes
2019-01-01 11:47:41 +02:00
Shlomi Noach
0be91ba0f2
trying something...
2019-01-01 11:39:43 +02:00
Shlomi Noach
255314927d
Updated go-mysql library, now with fix to decimal. May break datetime
2019-01-01 10:57:46 +02:00
Shlomi Noach
419794e16d
Merge branch 'master' into master
2018-12-25 07:20:47 +02:00
黄恒
29e3d48c36
Merge branch 'master' into master
2018-12-11 13:12:47 +08:00
Eduardo Nunes
5f15f8a539
Remove not needed call by grouping columnLists
...
There is no need to call `applyColumnTypes` more than once for the same
`databaseName` and `tableName`, it is just move the additional
`columnList` to the first call.
2018-11-01 14:43:57 +01:00
Eduardo Nunes
dfc8aa28f4
Ignore not available columns when applying column types
...
The given `columnLists` may not contain all columns available in the
given table. This patch prevents the code to fail. Before this patch the
code was panicking whenever it was trying to set a value into a `nil`
object, e.g. `columnList.GetColumn('non-existant').Type = SomeType`.
For some reason the application was not completely failing but as a
side-effect any column after the non-existant column would never get its
Type properly set.
2018-11-01 14:33:11 +01:00
Shlomi Noach
60398e774f
Merge branch 'master' into add-dry-run-env
2018-10-02 14:08:08 +03:00
Shlomi Noach
6284a34823
Merge branch 'master' into ipv6
2018-10-02 08:12:15 +03:00
Shlomi Noach
7bd338faf7
Fix to retry logic in row copy
2018-09-20 10:49:21 +03:00
Jacob Bednarz
09a005d79e
Populate the env with dry run status
...
Updates the `applyEnvironmentVariables` function to populate whether or
not the current execution context is running as a dry run or not which
can then be used in hooks.
2018-09-18 14:41:26 +10:00
Shlomi Noach
04c0be6137
Merge branch 'master' into ipv6
2018-09-17 09:35:18 +03:00
Shlomi Noach
97c1e72dbe
Merge branch 'master' into minor_changes
2018-09-17 08:58:04 +03:00
Shlomi Noach
695e32d41b
Merge branch 'master' into dev_copyrow_hangs
2018-09-17 08:42:03 +03:00
Shlomi Noach
959d1af211
support ipv6 without port
2018-09-16 11:52:59 +03:00
Shlomi Noach
a7cfaa4d33
added testing
2018-09-16 11:48:15 +03:00
Shlomi Noach
49b80df27b
Parsing ipv6 addresses
2018-09-16 11:44:52 +03:00
Shlomi Noach
32f1cf1dbe
Support for GCP (Google Cloud Platform)
2018-09-02 09:51:40 +03:00
钟越
ef686a4e7b
Support database with underscore
2018-08-31 09:31:58 +08:00
wangfei
c7dff99a19
fix copyRowsFunc hangs bug
2018-07-26 03:02:12 +08:00
MOON_CLJ
14eda7efe0
fix GetReplicationLag not used args
2018-05-23 13:13:25 +08:00
MOON_CLJ
21d455013e
minor changes and typo correction
2018-05-23 13:13:25 +08:00
Shlomi Noach
db871b42c4
Support for GENERATED (aka virtual) columns
2018-05-22 12:36:52 +03:00
Shlomi Noach
dfeefed5dc
avoid prepared statements for changelog table
2018-05-20 13:46:33 +03:00
Shlomi Noach
46bead0369
Merge branch 'master' into fClose-Fix
2018-05-08 07:53:56 +03:00
Shlomi Noach
24ed4ce5e6
Merge branch 'master' into reject-rename-table
2018-05-07 08:04:47 +03:00
Shlomi Noach
fb00a13871
Rejecting RENAME TO|AS
2018-05-06 11:19:03 +03:00
Cyrill Schumacher
c6a6a533a1
go/base: Do not ignore f.Close error
2018-05-04 08:22:54 +02:00
Atsushi Ishibashi
99fcbc35f3
Fix custom type assignment
2018-05-04 01:45:14 +09:00
Shlomi Noach
160e7a11f5
Enforcing binlog_row_image=FULL
2018-04-24 05:56:03 -07:00
Shlomi Noach
3dc4930ae3
Merge branch 'master' into support-aliyun-rds
2018-04-16 14:17:03 +03:00
zj118228
d3b98d9e74
Refactor the code and remove the global variable of
...
support-aliyun-rds branch.
2018-03-29 12:20:41 +08:00
Shlomi Noach
b3f599acd8
Merge branch 'master' into support-aliyun-rds
2018-03-27 14:23:51 +03:00
Kurt Kotzur
15e7417fc5
update comment
2018-03-19 12:29:49 -07:00
Kurt Kotzur
117b197b2a
Obey defaultNumRetries in retryOperationWithExponentialBackoff; name max interval flag more generically ( #2 )
...
* rename flags, obey defaultNumRetries
* capitalization fixes
* fix flag description typo
* fix sleep algorithm
2018-03-19 12:26:46 -07:00
Kurt Kotzur
628983f78f
🎨
2018-03-16 12:42:38 -07:00
Kurt Kotzur
64f66c4abb
add back nonFatalHint, fix type mismatches
2018-03-16 10:54:37 -07:00
Kurt Kotzur
bd62b19b0b
remove nonFatalHint argument from retryOperation; add retryOperationWithExponentialBackoff and conditionally use it
2018-03-15 17:41:15 -07:00
Kurt Kotzur
3f0443833d
add flags for enabling exponential backoff and maximum backoff interval
2018-03-15 17:40:15 -07:00
Shlomi Noach
710abd5a5b
Fixed test
2018-03-12 16:20:46 +02:00
Shlomi Noach
2311df412d
gbk support, simplified
2018-03-12 16:17:53 +02:00
Shlomi Noach
bf01b31795
Merge branch 'master' into support-aliyun-rds
2018-03-07 16:39:55 +02:00
Shlomi Noach
2209136258
rename parameters
2018-03-07 16:30:02 +02:00
Shlomi Noach
1a882c6eb5
Merge branch 'master' into gbk_encoding
2018-02-27 14:06:59 +02:00
Shlomi Noach
e2f6f4bc36
Merge branch 'master' into gh-ost-ci-env
2018-02-27 13:26:56 +02:00
Shlomi Noach
9ae9cef17a
support quoted values in interactive commands
2018-02-25 19:17:37 +02:00
Shlomi Noach
9b0efd0d93
delegating to 127.0.0.1
2018-02-11 14:58:16 +02:00
zj118228
6f3d54a136
Aliyun RDS hide some mysql variables for secrity so gh-ost will get
...
ilegal values. Add --aliyun-rds flag to avoid getting them.
2018-01-23 14:44:22 +08:00
ceshihao
330650baa9
rename includeMultibyteCharset -> includeRiskCharset, multibyteCharset -> riskCharset
2018-01-17 00:15:50 +08:00
ceshihao
14315ee2cc
resolve merge conflict
2018-01-16 23:43:42 +08:00
ceshihao
346138772c
add an option --include-multibyte-charset
2018-01-16 23:43:33 +08:00
ceshihao
35becabf3c
support gbk encoding
2018-01-16 23:43:09 +08:00
Shlomi Noach
a01e3639a2
Merge branch 'master' into go-mysql-ignore-pass
2018-01-14 10:08:58 +02:00
Shlomi Noach
fcda14634d
go-mysql library to avoid printing password in cleartext
2018-01-11 14:51:14 +02:00
Shlomi Noach
02579075d0
Call the throttler.Teardown
2018-01-11 08:03:09 +02:00
Nikhil Mathew
e82d5633f9
Close the binlog syncer
2017-12-18 11:03:37 -08:00
Nikhil Mathew
ec6ceffbcb
Pass in a migrationContext UUID for a migration specific connections cache
2017-12-14 15:53:29 -08:00
Nikhil Mathew
fac1ba7026
Throttler teardown
2017-12-14 15:14:13 -08:00
Shlomi Noach
7ef284a3ad
removed redundant CanStopStreaming func
2017-12-11 08:02:32 +02:00
Shlomi Noach
0530b09700
Merge branch 'master' into nm-refactor-migration-context
2017-12-07 13:59:12 +02:00
Shlomi Noach
58c381f0a0
some cleanup
2017-11-21 09:14:04 +02:00
Shlomi Noach
203ea6c2cd
refactor, simplifid code
2017-11-20 10:24:28 +02:00
Shlomi Noach
3898d49f6c
supporting update to columns of migration key
2017-11-20 08:17:20 +02:00
Nikhil Mathew
538833ea84
Merge branch 'master' into nm-refactor-migration-context
2017-11-14 10:29:49 -08:00
Nikhil Mathew
2ad65ee1d7
Refactor usages of GetDB to use mysql
2017-11-10 15:44:22 -08:00
Nikhil Mathew
bb4a0be4c6
Add GetDB to mysql.GetDB
2017-11-10 15:41:37 -08:00
Nikhil Mathew
338a439c91
Remove GetDB from MigrationContext
2017-11-10 15:40:50 -08:00
Shlomi Noach
8a59d7e823
failing on PK value change
2017-11-08 11:11:17 +02:00
Josh Soref
15b8d5769b
spelling: unable
2017-11-08 01:02:47 +00:00
Josh Soref
6bc04eee62
spelling: thing
2017-11-08 01:02:32 +00:00
Josh Soref
cb97d6854c
spelling: synchronously
2017-11-08 01:02:21 +00:00
Josh Soref
94dfa0ed24
spelling: specified
2017-11-08 01:01:51 +00:00
Josh Soref
3196c890f5
spelling: running
2017-11-08 01:01:39 +00:00
Josh Soref
b3be5352c5
spelling: rotates
2017-11-08 00:56:14 +00:00
Josh Soref
4c93b7ac6d
spelling: response
2017-11-08 00:55:09 +00:00
Josh Soref
961e2409f6
spelling: proceeding
2017-11-08 00:50:56 +00:00
Josh Soref
6a8d97c13f
spelling: progress
2017-11-08 00:50:48 +00:00
Josh Soref
867e6a28b3
spelling: merely
2017-11-08 00:49:51 +00:00
Josh Soref
e2df0b80eb
spelling: iterating
2017-11-08 00:49:23 +00:00
Josh Soref
2b430e3cb6
spelling: ignoring
2017-11-08 00:49:06 +00:00
Josh Soref
ed35b75cc9
spelling: identified
2017-11-08 00:48:53 +00:00
Josh Soref
e608c16567
spelling: explicitly
2017-11-08 00:47:54 +00:00
Josh Soref
291c6d3594
spelling: entirely
2017-11-08 00:47:36 +00:00
Josh Soref
9bb6959d22
spelling: empty
2017-11-08 00:47:24 +00:00
Josh Soref
418462a6ca
spelling: dropping
2017-11-08 00:47:15 +00:00
Josh Soref
1ae6489afc
spelling: detached
2017-11-08 00:46:26 +00:00
Josh Soref
8901e6eec2
spelling: decision
2017-11-08 00:46:04 +00:00
Josh Soref
84be0562a0
spelling: coordinate
2017-11-08 00:45:47 +00:00
Josh Soref
e99333e576
spelling: conversion
2017-11-08 00:45:31 +00:00
Josh Soref
d29993ffda
spelling: changelog
2017-11-08 00:45:19 +00:00
Josh Soref
4dda9cf930
spelling: aggressive
2017-11-08 00:44:30 +00:00
Akshay Chhajed
05fc358ed4
send -help output to stdout
2017-11-01 00:24:09 +05:30
Akshay Chhajed
ccb7654235
Improved connection type logging
2017-10-29 19:53:32 +05:30
Shlomi Noach
087a6cc0cb
Merge branch 'master' into touch-postpone-flag-file
2017-10-17 16:02:11 +03:00
Nikhil Mathew
f06e23282e
Merge branch 'master' into nm-refactor-migration-context
2017-10-02 10:31:57 -07:00
Shlomi Noach
aff7413624
clearer error message when not able to create postpone-cut-over-flag-file
2017-10-02 16:00:58 +03:00
Shlomi Noach
c2186db527
Merge branch 'master' into touch-postpone-flag-file
2017-10-02 15:55:21 +03:00
Shlomi Noach
9890e66755
Merge pull request #491 from zmoazeni/enable-extra-port
...
Allow gh-ost to modify the server using extra port
2017-10-02 14:26:59 +03:00
Nikhil Mathew
84bdfdb1ad
Cache DB connection pools on the migrationContext where applicable
2017-09-22 16:06:06 -07:00
Nikhil Mathew
5788ab5347
CR Revisions: Make concurrent operations safe
2017-09-22 15:17:58 -07:00
dcadevil
351bfe1a24
Correct spelling for initiateStreaming() comment.
2017-09-21 11:36:27 +08:00
Zach Moazeni
df27c5b76f
Allow gh-ost to modify the server using extra port
...
Both Percona and Maria allow MySQL to be configured to listen on an extra port when their thread pool is enable.
* https://www.percona.com/doc/percona-server/5.7/performance/threadpool.html
* https://mariadb.com/kb/en/the-mariadb-library/thread-pool-in-mariadb-51-53/
This is valuable because if the table has a lot of traffic (read or write load), gh-ost can end up starving the thread pool as incomming connections are immediately blocked.
By using gh-ost on the extra port, MySQL locking will still behave the same, but MySQL will keep a dedicated thread for each gh-ost connection.
When doing this, it's important to inspect the extra-max-connections variable. Both Percona and Maria default to 1, so gh-ost may easily exceed with its threads.
An example local run using this
```
$ mysql -S /tmp/mysql_sandbox20393.sock -e "select @@global.port, @@global.extra_port"
+---------------+---------------------+
| @@global.port | @@global.extra_port |
+---------------+---------------------+
| 20393 | 30393 |
+---------------+---------------------+
./bin/gh-ost \
--initially-drop-ghost-table \
--initially-drop-old-table \
--assume-rbr \
--port="20395" \
--assume-master-host="127.0.0.1:30393" \
--max-load=Threads_running=25 \
--critical-load=Threads_running=1000 \
--chunk-size=1000 \
--max-lag-millis=1500 \
--user="gh-ost" \
--password="gh-ost" \
--database="test" \
--table="mytable" \
--verbose \
--alter="ADD mynewcol decimal(11,2) DEFAULT 0.0 NOT NULL" \
--exact-rowcount \
--concurrent-rowcount \
--default-retries=120 \
--panic-flag-file=/tmp/ghost.panic.flag \
--postpone-cut-over-flag-file=/tmp/ghost.postpone.flag \
--execute
```
2017-09-20 16:05:20 -04:00
Shlomi Noach
6362dbe51a
Clearer error message on config file read error
2017-09-10 08:20:01 +03:00
Shlomi Noach
a2847015d6
Merge branch 'master' into nm-refactor-migration-context
2017-09-10 08:08:39 +03:00
Shlomi Noach
8940649a32
Merge branch 'master' into checking-thresholds
2017-09-06 15:14:17 +03:00
Peng Sung
113e115c49
Update the main.go word spelling error
2017-09-06 15:12:09 +08:00
Shlomi Noach
78c1e0432f
Merge branch 'master' into checking-thresholds
2017-09-06 08:07:35 +03:00
Shlomi Noach
e2171e0162
Validating password length
2017-09-03 10:27:04 +03:00
Shlomi Noach
3437cf44c4
Validating shared key column types
2017-09-03 09:57:26 +03:00
Shlomi Noach
f2cd598376
Merge branch 'master' into autocommit
2017-09-03 07:55:13 +03:00
Christian Joudrey
2646a0a997
Remove unused acceptSignals in migrator
2017-09-02 21:59:09 -04:00
Shlomi Noach
a3ee81e116
Merge branch 'master' into autocommit
2017-08-30 11:40:56 +03:00
Shlomi Noach
10cafb71c0
gh-ost uses autocommit=1 for all connections
2017-08-30 11:38:22 +03:00
Nikhil Mathew
0a7e713e9f
Ensure cleanup happens, even on error
2017-08-28 15:53:47 -07:00
Nikhil Mathew
e4bb70df43
Use change from outbrain/golib PR
2017-08-28 15:19:05 -07:00
Nikhil Mathew
3b21f4db37
Close remaining DB connections
2017-08-28 14:05:15 -07:00
Shlomi Noach
c9f2da82f0
Merge branch 'master' into json-57-test
2017-08-27 08:01:02 +03:00
Shlomi Noach
2174588ebf
Supporting JSON types
2017-08-22 11:51:20 +03:00
Shlomi Noach
79e4f1cdbe
detect range end based on OFFSET
2017-08-21 08:12:41 +03:00
Nikhil Mathew
a1473dfb58
Stop infinite goroutines
2017-08-08 15:31:25 -07:00
Nikhil Mathew
7412f42371
Remove no longer needed acceptSignals func
2017-08-08 13:44:53 -07:00
Nikhil Mathew
b399533d4f
Simplify GoMySQLReader
2017-08-08 13:41:46 -07:00
Nikhil Mathew
982b8eede9
Refactor global migration context
2017-08-08 13:36:54 -07:00
Shlomi Noach
d6f1675f1b
Merge branch 'master' into interpolate-params
2017-07-26 10:01:41 +03:00
Shlomi Noach
d9a23a2212
interpolateParams=true: client side prepared statements
2017-07-25 09:21:30 +03:00
Arthur Neves
02359b2a79
Dont share the same context in the table name tests
2017-07-24 06:26:31 -04:00
Arthur Neves
843c74f960
Add a force-table-names command flag
...
usage example:
```
gh-ost ... --force-table-names=migration_tbl
```
The force-table-names flag will receive a table name and use that name
when creating the _del, _gho, _ghc tables.
For instance for that example above, the tables would be:
_migration_tbl_del
_migration_tbl_gho
_migration_tbl_ghc
[fixes #399 ]
2017-07-21 15:37:57 -04:00
Shlomi Noach
e2c14f5aba
Merge branch 'master' into dynamic-dml-batch-size
2017-07-20 16:47:22 +03:00
Shlomi Noach
260374aeb9
Merge branch 'master' into long-table-names
2017-07-20 16:46:41 +03:00
Shlomi Noach
de0ebcbbb0
documentation
2017-07-19 16:48:22 +03:00
Shlomi Noach
6da0b8af3d
Dynamic DML batch size; apadting buffer size to match
2017-07-19 16:44:18 +03:00
Shlomi Noach
39e73e37d0
testing context.TimestampOldTable
2017-07-19 15:22:46 +03:00
Shlomi Noach
168676bd9d
ghost/changelog/del table names support max characters limit
2017-07-19 15:14:03 +03:00
Shlomi Noach
4a3a5a1ce1
Merge branch 'master' into case-insensitive-columns
2017-07-16 06:51:39 +03:00
Shlomi Noach
effae08fbf
supporting change of column name-case
2017-07-12 12:33:32 +03:00
Shlomi Noach
e6dd2c90d2
responding to errors in rowcopy
2017-07-11 13:39:11 +03:00
Shlomi Noach
81403699b7
Merge branch 'master' into throttle-http-freno-aware
2017-07-05 10:16:56 +03:00
Shlomi Noach
a8932ee1b8
freno-aware http-throttler
2017-07-04 10:28:09 +03:00
Shlomi Noach
f9788a61ed
Merge branch 'master' into dml-batch-size-cap
2017-07-04 08:54:49 +03:00
Shlomi Noach
095863b774
dml-batch-size capped at 1000
2017-07-04 08:50:25 +03:00
Shlomi Noach
36bcc031ce
Merge branch 'master' into fix-variables-spelling
2017-06-29 20:53:49 +03:00
Shlomi Noach
e8b384e43c
supporting greater batch size
2017-06-28 19:35:55 +03:00
Shlomi Noach
cb3c7f3e20
Merge branch 'master' into touch-postpone-flag-file
2017-06-11 08:57:45 +03:00