octoleo/gh-ost
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
1,498 Commits 51 Branches 51 Tags 24 MiB
dc9a2e3f24
Commit Graph

590 Commits

Author SHA1 Message Date
Brandon Bodnar
dc599bb036 Remove unnecessary branching 2019-01-31 17:20:11 -06:00
Brandon Bodnar
4f9367e690 Fix casing for initialism 2019-01-31 17:00:13 -06:00
Brandon Bodnar
23617f287f Add initial support for ssl encryption connections to database servers. 
- Adding a command line option for users to enforce tls/ssl connections
  for the applier, inspector, and binlog reader.
- The user can optionally request server certificate verification through
  a command line option to specify the ca cert via a file path.
- Fixes an existing bug appending the timeout option to the singleton
  applier connection.
 2019-01-31 15:22:03 -06:00
Shlomi Noach
1671530300
Merge branch 'master' into hints-owner-token 2019-01-28 08:28:58 +02:00
Shlomi Noach
3161cd5823 added: --hooks-hint-owner, --hooks-hint-token 2019-01-24 13:06:20 +02:00
Shlomi Noach
93b88e262a
Auto-merged master into gh-ost/patch-1 on deployment 2019-01-23 08:13:55 +02:00
twotwotwo
afa108f8fa
inspect: remove redundant join conditions 
The TABLE_SCHEMA and TABLE_NAME are already guaranteed to have the same value in COLUMNS and UNIQUES because of the WHEREs in each query. Dropping it from the ON clause makes it complete much faster.

On (at least) MySQL 5.6 installs with thousands of tables, this query completes in a few seconds without the additional join conditions, but takes more than a minute with it.
 2019-01-22 12:26:55 -08:00
Shlomi Noach
809908cb1d
Merge branch 'master' into ipv6 2019-01-20 09:02:34 +02:00
Shlomi Noach
5f587c3262 Adding --force-named-panic option 2019-01-14 13:27:44 +02:00
Shlomi Noach
d32304520c
Merge branch 'master' into minor_changes 2019-01-14 10:45:27 +02:00
Shlomi Noach
7c17cee010 reverting some changes 2019-01-01 11:47:41 +02:00
Shlomi Noach
0be91ba0f2 trying something... 2019-01-01 11:39:43 +02:00
Shlomi Noach
255314927d Updated go-mysql library, now with fix to decimal. May break datetime 2019-01-01 10:57:46 +02:00
Shlomi Noach
419794e16d
Merge branch 'master' into master 2018-12-25 07:20:47 +02:00
黄恒
29e3d48c36
Merge branch 'master' into master 2018-12-11 13:12:47 +08:00
Eduardo Nunes
5f15f8a539 Remove not needed call by grouping columnLists 
There is no need to call `applyColumnTypes` more than once for the same
`databaseName` and `tableName`, it is just move the additional
`columnList` to the first call.
 2018-11-01 14:43:57 +01:00
Eduardo Nunes
dfc8aa28f4 Ignore not available columns when applying column types 
The given `columnLists` may not contain all columns available in the
given table. This patch prevents the code to fail. Before this patch the
code was panicking whenever it was trying to set a value into a `nil`
object, e.g. `columnList.GetColumn('non-existant').Type = SomeType`.

For some reason the application was not completely failing but as a
side-effect any column after the non-existant column would never get its
Type properly set.
 2018-11-01 14:33:11 +01:00
Shlomi Noach
60398e774f
Merge branch 'master' into add-dry-run-env 2018-10-02 14:08:08 +03:00
Shlomi Noach
6284a34823
Merge branch 'master' into ipv6 2018-10-02 08:12:15 +03:00
Shlomi Noach
7bd338faf7 Fix to retry logic in row copy 2018-09-20 10:49:21 +03:00
Jacob Bednarz
09a005d79e
Populate the env with dry run status 
Updates the `applyEnvironmentVariables` function to populate whether or
not the current execution context is running as a dry run or not which
can then be used in hooks.
 2018-09-18 14:41:26 +10:00
Shlomi Noach
04c0be6137
Merge branch 'master' into ipv6 2018-09-17 09:35:18 +03:00
Shlomi Noach
97c1e72dbe
Merge branch 'master' into minor_changes 2018-09-17 08:58:04 +03:00
Shlomi Noach
695e32d41b
Merge branch 'master' into dev_copyrow_hangs 2018-09-17 08:42:03 +03:00
Shlomi Noach
959d1af211 support ipv6 without port 2018-09-16 11:52:59 +03:00
Shlomi Noach
a7cfaa4d33 added testing 2018-09-16 11:48:15 +03:00
Shlomi Noach
49b80df27b Parsing ipv6 addresses 2018-09-16 11:44:52 +03:00
Shlomi Noach
32f1cf1dbe Support for GCP (Google Cloud Platform) 2018-09-02 09:51:40 +03:00
钟越
ef686a4e7b Support database with underscore 2018-08-31 09:31:58 +08:00
wangfei
c7dff99a19 fix copyRowsFunc hangs bug 2018-07-26 03:02:12 +08:00
MOON_CLJ
14eda7efe0 fix GetReplicationLag not used args 2018-05-23 13:13:25 +08:00
MOON_CLJ
21d455013e minor changes and typo correction 2018-05-23 13:13:25 +08:00
Shlomi Noach
db871b42c4 Support for GENERATED (aka virtual) columns 2018-05-22 12:36:52 +03:00
Shlomi Noach
dfeefed5dc avoid prepared statements for changelog table 2018-05-20 13:46:33 +03:00
Shlomi Noach
46bead0369
Merge branch 'master' into fClose-Fix 2018-05-08 07:53:56 +03:00
Shlomi Noach
24ed4ce5e6
Merge branch 'master' into reject-rename-table 2018-05-07 08:04:47 +03:00
Shlomi Noach
fb00a13871 Rejecting RENAME TO|AS 2018-05-06 11:19:03 +03:00
Cyrill Schumacher
c6a6a533a1 go/base: Do not ignore f.Close error 2018-05-04 08:22:54 +02:00
Atsushi Ishibashi
99fcbc35f3 Fix custom type assignment 2018-05-04 01:45:14 +09:00
Shlomi Noach
160e7a11f5 Enforcing binlog_row_image=FULL 2018-04-24 05:56:03 -07:00
Shlomi Noach
3dc4930ae3
Merge branch 'master' into support-aliyun-rds 2018-04-16 14:17:03 +03:00
zj118228
d3b98d9e74 Refactor the code and remove the global variable of 
support-aliyun-rds branch.
 2018-03-29 12:20:41 +08:00
Shlomi Noach
b3f599acd8
Merge branch 'master' into support-aliyun-rds 2018-03-27 14:23:51 +03:00
Kurt Kotzur
15e7417fc5
update comment 2018-03-19 12:29:49 -07:00
Kurt Kotzur
117b197b2a
Obey defaultNumRetries in retryOperationWithExponentialBackoff; name max interval flag more generically (#2) 
* rename flags, obey defaultNumRetries

* capitalization fixes

* fix flag description typo

* fix sleep algorithm
 2018-03-19 12:26:46 -07:00
Kurt Kotzur
628983f78f
🎨 2018-03-16 12:42:38 -07:00
Kurt Kotzur
64f66c4abb
add back nonFatalHint, fix type mismatches 2018-03-16 10:54:37 -07:00
Kurt Kotzur
bd62b19b0b
remove nonFatalHint argument from retryOperation; add retryOperationWithExponentialBackoff and conditionally use it 2018-03-15 17:41:15 -07:00
Kurt Kotzur
3f0443833d
add flags for enabling exponential backoff and maximum backoff interval 2018-03-15 17:40:15 -07:00
Shlomi Noach
710abd5a5b Fixed test 2018-03-12 16:20:46 +02:00
Shlomi Noach
2311df412d gbk support, simplified 2018-03-12 16:17:53 +02:00
Shlomi Noach
bf01b31795
Merge branch 'master' into support-aliyun-rds 2018-03-07 16:39:55 +02:00
Shlomi Noach
2209136258 rename parameters 2018-03-07 16:30:02 +02:00
Shlomi Noach
1a882c6eb5
Merge branch 'master' into gbk_encoding 2018-02-27 14:06:59 +02:00
Shlomi Noach
e2f6f4bc36
Merge branch 'master' into gh-ost-ci-env 2018-02-27 13:26:56 +02:00
Shlomi Noach
9ae9cef17a support quoted values in interactive commands 2018-02-25 19:17:37 +02:00
Shlomi Noach
9b0efd0d93 delegating to 127.0.0.1 2018-02-11 14:58:16 +02:00
zj118228
6f3d54a136 Aliyun RDS hide some mysql variables for secrity so gh-ost will get 
ilegal values. Add --aliyun-rds flag to avoid getting them.
 2018-01-23 14:44:22 +08:00
ceshihao
330650baa9 rename includeMultibyteCharset -> includeRiskCharset, multibyteCharset -> riskCharset 2018-01-17 00:15:50 +08:00
ceshihao
14315ee2cc resolve merge conflict 2018-01-16 23:43:42 +08:00
ceshihao
346138772c add an option --include-multibyte-charset 2018-01-16 23:43:33 +08:00
ceshihao
35becabf3c support gbk encoding 2018-01-16 23:43:09 +08:00
Shlomi Noach
a01e3639a2
Merge branch 'master' into go-mysql-ignore-pass 2018-01-14 10:08:58 +02:00
Shlomi Noach
fcda14634d go-mysql library to avoid printing password in cleartext 2018-01-11 14:51:14 +02:00
Shlomi Noach
02579075d0 Call the throttler.Teardown 2018-01-11 08:03:09 +02:00
Nikhil Mathew
e82d5633f9 Close the binlog syncer 2017-12-18 11:03:37 -08:00
Nikhil Mathew
ec6ceffbcb Pass in a migrationContext UUID for a migration specific connections cache 2017-12-14 15:53:29 -08:00
Nikhil Mathew
fac1ba7026 Throttler teardown 2017-12-14 15:14:13 -08:00
Shlomi Noach
7ef284a3ad removed redundant CanStopStreaming func 2017-12-11 08:02:32 +02:00
Shlomi Noach
0530b09700
Merge branch 'master' into nm-refactor-migration-context 2017-12-07 13:59:12 +02:00
Shlomi Noach
58c381f0a0 some cleanup 2017-11-21 09:14:04 +02:00
Shlomi Noach
203ea6c2cd refactor, simplifid code 2017-11-20 10:24:28 +02:00
Shlomi Noach
3898d49f6c supporting update to columns of migration key 2017-11-20 08:17:20 +02:00
Nikhil Mathew
538833ea84
Merge branch 'master' into nm-refactor-migration-context 2017-11-14 10:29:49 -08:00
Nikhil Mathew
2ad65ee1d7 Refactor usages of GetDB to use mysql 2017-11-10 15:44:22 -08:00
Nikhil Mathew
bb4a0be4c6 Add GetDB to mysql.GetDB 2017-11-10 15:41:37 -08:00
Nikhil Mathew
338a439c91 Remove GetDB from MigrationContext 2017-11-10 15:40:50 -08:00
Shlomi Noach
8a59d7e823 failing on PK value change 2017-11-08 11:11:17 +02:00
Josh Soref
15b8d5769b spelling: unable 2017-11-08 01:02:47 +00:00
Josh Soref
6bc04eee62 spelling: thing 2017-11-08 01:02:32 +00:00
Josh Soref
cb97d6854c spelling: synchronously 2017-11-08 01:02:21 +00:00
Josh Soref
94dfa0ed24 spelling: specified 2017-11-08 01:01:51 +00:00
Josh Soref
3196c890f5 spelling: running 2017-11-08 01:01:39 +00:00
Josh Soref
b3be5352c5 spelling: rotates 2017-11-08 00:56:14 +00:00
Josh Soref
4c93b7ac6d spelling: response 2017-11-08 00:55:09 +00:00
Josh Soref
961e2409f6 spelling: proceeding 2017-11-08 00:50:56 +00:00
Josh Soref
6a8d97c13f spelling: progress 2017-11-08 00:50:48 +00:00
Josh Soref
867e6a28b3 spelling: merely 2017-11-08 00:49:51 +00:00
Josh Soref
e2df0b80eb spelling: iterating 2017-11-08 00:49:23 +00:00
Josh Soref
2b430e3cb6 spelling: ignoring 2017-11-08 00:49:06 +00:00
Josh Soref
ed35b75cc9 spelling: identified 2017-11-08 00:48:53 +00:00
Josh Soref
e608c16567 spelling: explicitly 2017-11-08 00:47:54 +00:00
Josh Soref
291c6d3594 spelling: entirely 2017-11-08 00:47:36 +00:00
Josh Soref
9bb6959d22 spelling: empty 2017-11-08 00:47:24 +00:00
Josh Soref
418462a6ca spelling: dropping 2017-11-08 00:47:15 +00:00
Josh Soref
1ae6489afc spelling: detached 2017-11-08 00:46:26 +00:00
Josh Soref
8901e6eec2 spelling: decision 2017-11-08 00:46:04 +00:00
Josh Soref
84be0562a0 spelling: coordinate 2017-11-08 00:45:47 +00:00
Josh Soref
e99333e576 spelling: conversion 2017-11-08 00:45:31 +00:00
Josh Soref
d29993ffda spelling: changelog 2017-11-08 00:45:19 +00:00
Josh Soref
4dda9cf930 spelling: aggressive 2017-11-08 00:44:30 +00:00
Akshay Chhajed
05fc358ed4 send -help output to stdout 2017-11-01 00:24:09 +05:30
Akshay Chhajed
ccb7654235 Improved connection type logging 2017-10-29 19:53:32 +05:30
Shlomi Noach
087a6cc0cb Merge branch 'master' into touch-postpone-flag-file 2017-10-17 16:02:11 +03:00
Nikhil Mathew
f06e23282e Merge branch 'master' into nm-refactor-migration-context 2017-10-02 10:31:57 -07:00
Shlomi Noach
aff7413624 clearer error message when not able to create postpone-cut-over-flag-file 2017-10-02 16:00:58 +03:00
Shlomi Noach
c2186db527 Merge branch 'master' into touch-postpone-flag-file 2017-10-02 15:55:21 +03:00
Shlomi Noach
9890e66755 Merge pull request #491 from zmoazeni/enable-extra-port 
Allow gh-ost to modify the server using extra port
 2017-10-02 14:26:59 +03:00
Nikhil Mathew
84bdfdb1ad Cache DB connection pools on the migrationContext where applicable 2017-09-22 16:06:06 -07:00
Nikhil Mathew
5788ab5347 CR Revisions: Make concurrent operations safe 2017-09-22 15:17:58 -07:00
dcadevil
351bfe1a24 Correct spelling for initiateStreaming() comment. 2017-09-21 11:36:27 +08:00
Zach Moazeni
df27c5b76f Allow gh-ost to modify the server using extra port 
Both Percona and Maria allow MySQL to be configured to listen on an extra port when their thread pool is enable.

* https://www.percona.com/doc/percona-server/5.7/performance/threadpool.html
* https://mariadb.com/kb/en/the-mariadb-library/thread-pool-in-mariadb-51-53/

This is valuable because if the table has a lot of traffic (read or write load), gh-ost can end up starving the thread pool as incomming connections are immediately blocked.

By using gh-ost on the extra port, MySQL locking will still behave the same, but MySQL will keep a dedicated thread for each gh-ost connection.

When doing this, it's important to inspect the extra-max-connections variable. Both Percona and Maria default to 1, so gh-ost may easily exceed with its threads.

An example local run using this

```
$ mysql -S /tmp/mysql_sandbox20393.sock -e "select @@global.port, @@global.extra_port"
+---------------+---------------------+
| @@global.port | @@global.extra_port |
+---------------+---------------------+
|         20393 |               30393 |
+---------------+---------------------+

./bin/gh-ost \
--initially-drop-ghost-table \
--initially-drop-old-table \
--assume-rbr \
--port="20395" \
--assume-master-host="127.0.0.1:30393" \
--max-load=Threads_running=25 \
--critical-load=Threads_running=1000 \
--chunk-size=1000 \
--max-lag-millis=1500 \
--user="gh-ost" \
--password="gh-ost" \
--database="test" \
--table="mytable" \
--verbose \
--alter="ADD mynewcol decimal(11,2) DEFAULT 0.0 NOT NULL" \
--exact-rowcount \
--concurrent-rowcount \
--default-retries=120 \
--panic-flag-file=/tmp/ghost.panic.flag \
--postpone-cut-over-flag-file=/tmp/ghost.postpone.flag \
--execute
```
 2017-09-20 16:05:20 -04:00
Shlomi Noach
6362dbe51a Clearer error message on config file read error 2017-09-10 08:20:01 +03:00
Shlomi Noach
a2847015d6 Merge branch 'master' into nm-refactor-migration-context 2017-09-10 08:08:39 +03:00
Shlomi Noach
8940649a32 Merge branch 'master' into checking-thresholds 2017-09-06 15:14:17 +03:00
Peng Sung
113e115c49 Update the main.go word spelling error 2017-09-06 15:12:09 +08:00
Shlomi Noach
78c1e0432f Merge branch 'master' into checking-thresholds 2017-09-06 08:07:35 +03:00
Shlomi Noach
e2171e0162 Validating password length 2017-09-03 10:27:04 +03:00
Shlomi Noach
3437cf44c4 Validating shared key column types 2017-09-03 09:57:26 +03:00
Shlomi Noach
f2cd598376 Merge branch 'master' into autocommit 2017-09-03 07:55:13 +03:00
Christian Joudrey
2646a0a997
Remove unused acceptSignals in migrator 2017-09-02 21:59:09 -04:00
Shlomi Noach
a3ee81e116 Merge branch 'master' into autocommit 2017-08-30 11:40:56 +03:00
Shlomi Noach
10cafb71c0 gh-ost uses autocommit=1 for all connections 2017-08-30 11:38:22 +03:00
Nikhil Mathew
0a7e713e9f Ensure cleanup happens, even on error 2017-08-28 15:53:47 -07:00
Nikhil Mathew
e4bb70df43 Use change from outbrain/golib PR 2017-08-28 15:19:05 -07:00
Nikhil Mathew
3b21f4db37 Close remaining DB connections 2017-08-28 14:05:15 -07:00
Shlomi Noach
c9f2da82f0 Merge branch 'master' into json-57-test 2017-08-27 08:01:02 +03:00
Shlomi Noach
2174588ebf Supporting JSON types 2017-08-22 11:51:20 +03:00
Shlomi Noach
79e4f1cdbe detect range end based on OFFSET 2017-08-21 08:12:41 +03:00
Nikhil Mathew
a1473dfb58 Stop infinite goroutines 2017-08-08 15:31:25 -07:00
Nikhil Mathew
7412f42371 Remove no longer needed acceptSignals func 2017-08-08 13:44:53 -07:00
Nikhil Mathew
b399533d4f Simplify GoMySQLReader 2017-08-08 13:41:46 -07:00
Nikhil Mathew
982b8eede9 Refactor global migration context 2017-08-08 13:36:54 -07:00
Shlomi Noach
d6f1675f1b Merge branch 'master' into interpolate-params 2017-07-26 10:01:41 +03:00
Shlomi Noach
d9a23a2212 interpolateParams=true: client side prepared statements 2017-07-25 09:21:30 +03:00
Arthur Neves
02359b2a79
Dont share the same context in the table name tests 2017-07-24 06:26:31 -04:00
Arthur Neves
843c74f960
Add a force-table-names command flag 
usage example:
```
gh-ost  ...  --force-table-names=migration_tbl
```

The force-table-names flag will receive a table name and use that name
when creating the _del, _gho, _ghc tables.
For instance for that example above, the tables would be:

_migration_tbl_del
_migration_tbl_gho
_migration_tbl_ghc

[fixes #399]
 2017-07-21 15:37:57 -04:00
Shlomi Noach
e2c14f5aba Merge branch 'master' into dynamic-dml-batch-size 2017-07-20 16:47:22 +03:00
Shlomi Noach
260374aeb9 Merge branch 'master' into long-table-names 2017-07-20 16:46:41 +03:00
Shlomi Noach
de0ebcbbb0 documentation 2017-07-19 16:48:22 +03:00
Shlomi Noach
6da0b8af3d Dynamic DML batch size; apadting buffer size to match 2017-07-19 16:44:18 +03:00
Shlomi Noach
39e73e37d0 testing context.TimestampOldTable 2017-07-19 15:22:46 +03:00
Shlomi Noach
168676bd9d ghost/changelog/del table names support max characters limit 2017-07-19 15:14:03 +03:00
Shlomi Noach
4a3a5a1ce1 Merge branch 'master' into case-insensitive-columns 2017-07-16 06:51:39 +03:00
Shlomi Noach
effae08fbf supporting change of column name-case 2017-07-12 12:33:32 +03:00
Shlomi Noach
e6dd2c90d2 responding to errors in rowcopy 2017-07-11 13:39:11 +03:00
Shlomi Noach
81403699b7 Merge branch 'master' into throttle-http-freno-aware 2017-07-05 10:16:56 +03:00
Shlomi Noach
a8932ee1b8 freno-aware http-throttler 2017-07-04 10:28:09 +03:00
Shlomi Noach
f9788a61ed Merge branch 'master' into dml-batch-size-cap 2017-07-04 08:54:49 +03:00
Shlomi Noach
095863b774 dml-batch-size capped at 1000 2017-07-04 08:50:25 +03:00