Commit Graph

167 Commits

Author SHA1 Message Date
Tim Vaillancourt
0918bab29b
Add context/timeout to HTTP throttle check (#1131)
* Add context/timeout to HTTP throttle check

* Dont run `.GetThrottleHTTPInterval()` on every loop

* Update help message

* Var rename

* 2022

* Add timeout flag

* Add unix/tcp server commands, use ParseInt() for string->int64

* Var rename

* Re-check http timeout on every loop iteration

* Remove stale comment

* Make throttle interval idempotent

* var rename

* Usage grammar

* Make http timeout idempotent too

* Parse time.Duration once

* Move timeout to NewThrottler

* Help update

* Set User-Agent header

* Re-add newline

Co-authored-by: dm-2 <45519614+dm-2@users.noreply.github.com>
2022-07-06 23:56:07 +02:00
Tim Vaillancourt
ed46138c06
Add golangci-lint CI action, fix gosimple, govet + unused lint errors (#1127)
* Add `golangci-lint`, fix `gosimple`, `govet` and `unused` linter complaints

* Go 1.16

* Update copyright dates
2022-05-31 21:23:39 +02:00
dm-2
94ffadf101
Merge branch 'master' into customize_status_hook_interval 2022-02-07 11:51:42 +00:00
Arthur Schreiber
997c5b8bff Reduce the minimal chunk size from 100 to 10. 2022-02-07 11:23:39 +00:00
Tyler Knodell
fbb53f4b04 Add flag to customize the interval which the onStatus hook is called 2022-02-01 08:54:55 -07:00
Tim Vaillancourt
47d49c6b92
Add go mod (#935)
* Add a go.mod file

* run go mod vendor again

* Move to a well-supported ini file reader

* Remove GO111MODULE=off

* Use go 1.16

* Rename github.com/outbrain/golib -> github.com/openark/golib

* Remove *.go-e files

* Fix for `strconv.ParseInt: parsing "": invalid syntax` error

* Add test for '[osc]' section

Co-authored-by: Nate Wernimont <nate.wernimont@workiva.com>
2021-06-24 20:19:37 +02:00
Shlomi Noach
f19f101610
hooks: reporting GH_OST_ETA_SECONDS. ETA as part of migration context (#936)
* v1.1.0

* WIP: copying AUTO_INCREMENT value to ghost table
Initial commit: towards setting up a test suite

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* greping for 'expect_table_structure' content

* Adding simple test for 'expect_table_structure' scenario

* adding tests for AUTO_INCREMENT value after row deletes. Should initially fail

* clear event beforehand

* parsing AUTO_INCREMENT from alter query, reading AUTO_INCREMENT from original table, applying AUTO_INCREMENT value onto ghost table if applicable and user has not specified AUTO_INCREMENT in alter statement

* support GetUint64

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* minor update to test

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* adding test for user defined AUTO_INCREMENT statement

* Generated column as part of UNIQUE (or PRIMARY) KEY

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* skip analysis of generated column data type in unique key

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* All MySQL DBs limited to max 3 concurrent/idle connections

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* hooks: reporting GH_OST_ETA_SECONDS. ETA stored as part of migration context

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* GH_OST_ETA_NANOSECONDS

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* N/A denoted by negative value

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* ETAUnknown constant

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
2021-05-31 14:15:51 +02:00
Tim Vaillancourt
c71dbf9ef3
Copy auto increment (#967)
* v1.1.0

* WIP: copying AUTO_INCREMENT value to ghost table
Initial commit: towards setting up a test suite

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* greping for 'expect_table_structure' content

* Adding simple test for 'expect_table_structure' scenario

* adding tests for AUTO_INCREMENT value after row deletes. Should initially fail

* clear event beforehand

* parsing AUTO_INCREMENT from alter query, reading AUTO_INCREMENT from original table, applying AUTO_INCREMENT value onto ghost table if applicable and user has not specified AUTO_INCREMENT in alter statement

* support GetUint64

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* minor update to test

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* adding test for user defined AUTO_INCREMENT statement

Co-authored-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
2021-05-14 15:32:56 +02:00
Cathal Coffey
48ce0873de Store lastHeartbeatOnChangelogTime instead of CurrentHeartbeatLag 2021-02-03 10:03:45 +00:00
Cathal Coffey
a4218cd6f4 Progress should print HeartbeatLag 2021-01-31 18:27:34 +00:00
Cathal Coffey
8aee288fd7 Handle onChangelogHeartbeatEvent and update CurrentHeartbeatLag 2021-01-31 18:24:26 +00:00
Tim Vaillancourt
88e59dd2a3
Merge branch 'master' into master 2020-10-22 23:22:32 +02:00
Shlomi Noach
c9249f2b71 Updating and using AlterTableOptions 2020-07-23 11:38:05 +03:00
Justin Fudally
b02900ae06
Merge branch 'master' into logging-interface-contrib 2020-07-21 16:01:29 -05:00
Shuode Li
57955b968b
Merge branch 'master' into master 2020-05-06 00:15:57 -07:00
Justin Fudally
57cf5f3c90 add override to ignore http errors 2020-03-25 15:58:32 -05:00
Justin Fudally
46dabd338b go fmt 2020-03-25 15:48:19 -05:00
Justin Fudally
d31f3aac0c Add CLI flag for ignoring http errors 2020-03-25 15:47:56 -05:00
Shuode Li
ae762694fb Support Azure Database for MySQL. 2019-12-17 03:47:14 +00:00
Shlomi Noach
a36d6bee94
Merge branch 'logging-interface-contrib' into master 2019-12-11 07:03:37 +02:00
Abeyu M
f4676bf463 implement a logging interface 2019-10-07 11:10:36 -04:00
Shlomi Noach
271c7274a3 refactor progressPct into migrationContext 2019-10-07 06:59:56 +03:00
Shlomi Noach
7869889988 context, status and hooks: progressPct and CurrentLag 2019-10-06 17:08:35 +03:00
Shlomi Noach
7ae32f7184
Merge branch 'master' into skip-strict-mode 2019-03-17 23:49:13 -07:00
Shlomi Noach
766040d83f
Merge branch 'master' into named-panic 2019-03-06 12:47:06 +02:00
Shlomi Noach
ff17d2f844
Merge branch 'master' into named-panic 2019-02-25 14:00:12 +02:00
Rafe Kettler
b4da7cfbfb Improve SSL support
Related issue: https://github.com/github/gh-ost/issues/521

 - Add --ssl-cert and --ssl-key options to specify SSL public/private
   key files
 - Allow combining --ssl-allow-insecure with other --ssl* flags.
   `mysql.RegisterTLSConfig` allows combining the corresponding
   parameters in the `tls.Config` it receives, so gh-ost should
   allow this. I found being able to pass --ssl-allow-insecure along
   with --ssl-ca, --ssl-cert, and --ssl-key useful in testing.
 - Use the same TLS config everywhere. Since the CLI only supports
   a single set of --ssl* configuration parameters, this should be
   fine -- `mysql.RegisterTLSConfig` documentation indicates the
   TLS config given will not be modified, so it can safely be used
   in many goroutines provided we also do not modify it. The previous
   implementation did not work when the TLS config was duplicated,
   which happens when gh-ost walks up the replication chain trying
   to find the master. This is because, when the config is duplicated,
   we must call `RegisterTLSConfig` again with the new config. This
   config is exactly the same, so it's easiest to side-step the issue
   by registering the TLS config once and using it everywhere.
2019-02-22 10:33:19 -08:00
babinomec
7d5749b84a add --skip-strict-mode option 2019-02-14 15:58:49 +01:00
Shlomi Noach
9da2e21e03
Merge branch 'master' into add-tls-support 2019-02-10 11:27:36 +02:00
Brandon Bodnar
5b0dfb009c Wireup allowing insecure ssl 2019-02-04 16:21:25 -06:00
Matt Belisle
79df0d1c5d Adding --ssl-insecure flag 2019-02-04 14:46:08 -06:00
Brandon Bodnar
4f9367e690 Fix casing for initialism 2019-01-31 17:00:13 -06:00
Brandon Bodnar
23617f287f Add initial support for ssl encryption connections to database servers.
- Adding a command line option for users to enforce tls/ssl connections
  for the applier, inspector, and binlog reader.
- The user can optionally request server certificate verification through
  a command line option to specify the ca cert via a file path.
- Fixes an existing bug appending the timeout option to the singleton
  applier connection.
2019-01-31 15:22:03 -06:00
Shlomi Noach
3161cd5823 added: --hooks-hint-owner, --hooks-hint-token 2019-01-24 13:06:20 +02:00
Shlomi Noach
5f587c3262 Adding --force-named-panic option 2019-01-14 13:27:44 +02:00
Shlomi Noach
32f1cf1dbe Support for GCP (Google Cloud Platform) 2018-09-02 09:51:40 +03:00
Shlomi Noach
db871b42c4 Support for GENERATED (aka virtual) columns 2018-05-22 12:36:52 +03:00
Atsushi Ishibashi
99fcbc35f3 Fix custom type assignment 2018-05-04 01:45:14 +09:00
Shlomi Noach
3dc4930ae3
Merge branch 'master' into support-aliyun-rds 2018-04-16 14:17:03 +03:00
zj118228
d3b98d9e74 Refactor the code and remove the global variable of
support-aliyun-rds branch.
2018-03-29 12:20:41 +08:00
Kurt Kotzur
117b197b2a
Obey defaultNumRetries in retryOperationWithExponentialBackoff; name max interval flag more generically (#2)
* rename flags, obey defaultNumRetries

* capitalization fixes

* fix flag description typo

* fix sleep algorithm
2018-03-19 12:26:46 -07:00
Kurt Kotzur
628983f78f
🎨 2018-03-16 12:42:38 -07:00
Kurt Kotzur
64f66c4abb
add back nonFatalHint, fix type mismatches 2018-03-16 10:54:37 -07:00
Kurt Kotzur
3f0443833d
add flags for enabling exponential backoff and maximum backoff interval 2018-03-15 17:40:15 -07:00
zj118228
6f3d54a136 Aliyun RDS hide some mysql variables for secrity so gh-ost will get
ilegal values. Add --aliyun-rds flag to avoid getting them.
2018-01-23 14:44:22 +08:00
Nikhil Mathew
ec6ceffbcb Pass in a migrationContext UUID for a migration specific connections cache 2017-12-14 15:53:29 -08:00
Shlomi Noach
7ef284a3ad removed redundant CanStopStreaming func 2017-12-11 08:02:32 +02:00
Nikhil Mathew
2ad65ee1d7 Refactor usages of GetDB to use mysql 2017-11-10 15:44:22 -08:00
Nikhil Mathew
338a439c91 Remove GetDB from MigrationContext 2017-11-10 15:40:50 -08:00
Nikhil Mathew
f06e23282e Merge branch 'master' into nm-refactor-migration-context 2017-10-02 10:31:57 -07:00