Commit Graph

1513 Commits

Author SHA1 Message Date
dm-2
605104c9a2
Merge branch 'master' into fix-rpm-build-os 2022-02-25 14:15:39 +00:00
dm-2
522fbb849c fix: update build script to explicitly build RPMs for linux 2022-02-25 14:10:42 +00:00
dm-2
4142f5d07b
Merge pull request #1095 from timvaillancourt/inspector-connectionConfig-instanceKey-string
Use `.String()` for logging connection-config `InstanceKey`
2022-02-24 10:45:11 +00:00
dm-2
260e662197
Merge branch 'master' into inspector-connectionConfig-instanceKey-string 2022-02-24 10:34:19 +00:00
dm-2
3824eec0eb
Merge pull request #1096 from timvaillancourt/inspector-fix-needless-fmt-sprintf
Fix needless `fmt.Sprintf` call in `go/logic/inspector.go`
2022-02-24 10:34:12 +00:00
Tim Vaillancourt
1aa74bfe60 Fix needless fmt.Sprintf call in go/logic/inspector.go 2022-02-24 01:30:29 +01:00
Tim Vaillancourt
f6db40e4cd Use .String() for logging connection-config InstanceKey 2022-02-23 23:28:03 +01:00
Tim Vaillancourt
ac23094497
Merge branch 'github:master' into master 2022-02-22 00:57:58 +01:00
dm-2
0c7c9a52f3
Merge pull request #1090 from github/add-hooks-status-interval-docs
Add docs for `hooks-status-interval`
2022-02-07 16:29:50 +00:00
dm-2
d84c687d7c
Add docs for hooks-status-interval 2022-02-07 15:35:14 +00:00
dm-2
ebd5645989
Merge pull request #1083 from tknodell-recurly/customize_status_hook_interval
Add flag to customize the interval which the onStatus hook is called
2022-02-07 12:46:43 +00:00
dm-2
94ffadf101
Merge branch 'master' into customize_status_hook_interval 2022-02-07 11:51:42 +00:00
dm-2
dc9a2e3f24
Merge pull request #1087 from github/arthur/reduce-minimal-chunk-size
Reduce the minimal chunk size from `100` to `10`.
2022-02-07 11:51:08 +00:00
Arthur Schreiber
997c5b8bff Reduce the minimal chunk size from 100 to 10. 2022-02-07 11:23:39 +00:00
Tyler Knodell
fbb53f4b04 Add flag to customize the interval which the onStatus hook is called 2022-02-01 08:54:55 -07:00
dm-2
d6c83638d7
Merge pull request #1066 from EagleEyeJohn/documentation-update
Documentation update
2022-01-18 17:00:16 +00:00
dm-2
619d9d3f5e
Merge branch 'master' into documentation-update 2022-01-18 16:41:18 +00:00
dm-2
39bba9c92a
Merge pull request #1078 from github/fix-build-script
Update build script to generate releases
2022-01-18 16:32:01 +00:00
dm-2
09c35bd393 Update build script to generate releases 2022-01-18 16:15:36 +00:00
dm-2
1f780ae973
Merge pull request from GHSA-rrp4-2xx3-mv29
Security: fix vulnerability where `-database` parameter accepts arbitrary DSN strings
2022-01-18 15:59:32 +00:00
dm-2
8cd027c7f5
Merge branch 'master' into advisory-fix-1 2022-01-18 12:09:26 +00:00
dm-2
83413c339e Security: fix vulnerability where -database parameter accepts arbitrary DSN strings 2022-01-18 11:23:49 +00:00
John Nicholls
66fadc71d7
Merge branch 'master' into documentation-update 2022-01-18 09:27:40 +00:00
Rashiq
be644ebd43
Merge pull request #1076 from github/rashiq-update_gh_ost_hook_docs 2022-01-14 14:52:53 +01:00
Rashiq
0dc9092d5c
Update hooks.md 2022-01-14 02:49:16 +01:00
John Nicholls
2c13d814ac Improve command line flags documentation grammar 2021-12-15 16:37:39 +00:00
John Nicholls
47fd51f804 Document critical-load-hibernate-seconds 2021-12-15 16:34:05 +00:00
Ed Toro
e484824bbd
typo (#1011)
"or and" => "or an"
2021-07-27 17:38:50 +02:00
Tim Vaillancourt
a1862908c9
Use github.com/go-sql-driver 1.6.0 (#1006)
* Replace deprecated go-mysql library

* Use github.com/go-sql-driver 1.6.0

* go mod vendor

* go mod vendor again
2021-07-19 17:00:35 +02:00
Tim Vaillancourt
732a064231
Replace deprecated go-mysql library (#994) 2021-07-15 21:49:50 +02:00
Tim Vaillancourt
43d3afea5a Merge branch 'master' of https://github.com/timvaillancourt/gh-ost 2021-07-15 21:08:01 +02:00
Andrew Mason
6e1daf90ee
Check RowsAffected when applying DML events to get more accurate statistics (#844)
* Check RowsAffected when applying DML events to get more accurate statistics

Addresses #600.

When applying a DML event, check the RowsAffected on the `Result`
struct. Since all DML event queries are point queries, this will only
ever be 0 or 1. The applier then takes this value and multiplies by
the `rowsDelta` of the event, resulting in a properly-signed, accurate
row delta to use in the statistics.

If an error occurs here, log it, but do not surface this as an
actual error .. simply assume the DML affected a row and move on. It
will be inaccurate, but this is already the case.

* Fix import

* update wording to warning log message

Co-authored-by: Tim Vaillancourt <timvaillancourt@github.com>

Co-authored-by: Tim Vaillancourt <timvaillancourt@github.com>
2021-07-14 16:48:03 +02:00
Tim Vaillancourt
d726b20dda
Add CodeSQL actions workflow (#1005) 2021-07-13 20:16:33 +02:00
Tim Vaillancourt
b5a1fecb11 Add CodeSQL actions workflow 2021-07-13 16:30:58 +02:00
Tim Vaillancourt
47d49c6b92
Add go mod (#935)
* Add a go.mod file

* run go mod vendor again

* Move to a well-supported ini file reader

* Remove GO111MODULE=off

* Use go 1.16

* Rename github.com/outbrain/golib -> github.com/openark/golib

* Remove *.go-e files

* Fix for `strconv.ParseInt: parsing "": invalid syntax` error

* Add test for '[osc]' section

Co-authored-by: Nate Wernimont <nate.wernimont@workiva.com>
2021-06-24 20:19:37 +02:00
Tim Vaillancourt
aef2a69903
v1.1.2 (#990) 2021-06-17 15:33:20 +02:00
Dirkjan Bussink
40acde0222
Set the ServerName for TLS configuration (#988)
When TLS hostname validation used for the MySQL connection, the
ServerName property needs to be set so that it knows which name to
validate on the certificate.

Without this option and with InsecureSkipVerify set to false, validation
will error here with a fatal error otherwise:

```
FATAL tls: either ServerName or InsecureSkipVerify must be specified in the tls.Config
```
2021-06-17 14:51:43 +02:00
Shlomi Noach
9bc508f068
Enum to varchar (#963)
* v1.1.0

* WIP: copying AUTO_INCREMENT value to ghost table
Initial commit: towards setting up a test suite

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* greping for 'expect_table_structure' content

* Adding simple test for 'expect_table_structure' scenario

* adding tests for AUTO_INCREMENT value after row deletes. Should initially fail

* clear event beforehand

* parsing AUTO_INCREMENT from alter query, reading AUTO_INCREMENT from original table, applying AUTO_INCREMENT value onto ghost table if applicable and user has not specified AUTO_INCREMENT in alter statement

* support GetUint64

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* minor update to test

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* adding test for user defined AUTO_INCREMENT statement

* Generated column as part of UNIQUE (or PRIMARY) KEY

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* skip analysis of generated column data type in unique key

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* All MySQL DBs limited to max 3 concurrent/idle connections

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* hooks: reporting GH_OST_ETA_SECONDS. ETA stored as part of migration context

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* GH_OST_ETA_NANOSECONDS

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* N/A denoted by negative value

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* ETAUnknown constant

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* Convering enum to varchar

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* test: not null

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* first attempt at setting enum-to-string right

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* fix insert query

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* store enum values, use when populating

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* apply EnumValues to mapped column

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* fix compilation error

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* gofmt

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
2021-06-10 17:17:49 +02:00
Shlomi Noach
f19f101610
hooks: reporting GH_OST_ETA_SECONDS. ETA as part of migration context (#936)
* v1.1.0

* WIP: copying AUTO_INCREMENT value to ghost table
Initial commit: towards setting up a test suite

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* greping for 'expect_table_structure' content

* Adding simple test for 'expect_table_structure' scenario

* adding tests for AUTO_INCREMENT value after row deletes. Should initially fail

* clear event beforehand

* parsing AUTO_INCREMENT from alter query, reading AUTO_INCREMENT from original table, applying AUTO_INCREMENT value onto ghost table if applicable and user has not specified AUTO_INCREMENT in alter statement

* support GetUint64

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* minor update to test

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* adding test for user defined AUTO_INCREMENT statement

* Generated column as part of UNIQUE (or PRIMARY) KEY

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* skip analysis of generated column data type in unique key

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* All MySQL DBs limited to max 3 concurrent/idle connections

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* hooks: reporting GH_OST_ETA_SECONDS. ETA stored as part of migration context

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* GH_OST_ETA_NANOSECONDS

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* N/A denoted by negative value

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* ETAUnknown constant

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
2021-05-31 14:15:51 +02:00
Shlomi Noach
c41823ecc9
All MySQL DBs limited to max 3 concurrent/idle connections #15 (#931)
* v1.1.0

* WIP: copying AUTO_INCREMENT value to ghost table
Initial commit: towards setting up a test suite

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* greping for 'expect_table_structure' content

* Adding simple test for 'expect_table_structure' scenario

* adding tests for AUTO_INCREMENT value after row deletes. Should initially fail

* clear event beforehand

* parsing AUTO_INCREMENT from alter query, reading AUTO_INCREMENT from original table, applying AUTO_INCREMENT value onto ghost table if applicable and user has not specified AUTO_INCREMENT in alter statement

* support GetUint64

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* minor update to test

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* adding test for user defined AUTO_INCREMENT statement

* Generated column as part of UNIQUE (or PRIMARY) KEY

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* skip analysis of generated column data type in unique key

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* All MySQL DBs limited to max 3 concurrent/idle connections

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
2021-05-27 20:00:58 +02:00
Tim Vaillancourt
8f42dedef8
Add GO111MODULE=off to build.sh, use Golang 1.16 (#966)
* Add GO111MODULE=off to build.sh

* Use golang 1.16

* Update go version in README.md

* Add missing GO111MODULE=off

* Add missing GO111MODULE=off again

* Use go1.16.3 explicitly

* Use 1.16 for CI test

* Update min go version

* Use go 1.16.4
2021-05-25 13:22:28 +02:00
Shlomi Noach
36c669dd75
Generated column as part of UNIQUE (or PRIMARY) KEY (#919)
* v1.1.0

* WIP: copying AUTO_INCREMENT value to ghost table
Initial commit: towards setting up a test suite

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* greping for 'expect_table_structure' content

* Adding simple test for 'expect_table_structure' scenario

* adding tests for AUTO_INCREMENT value after row deletes. Should initially fail

* clear event beforehand

* parsing AUTO_INCREMENT from alter query, reading AUTO_INCREMENT from original table, applying AUTO_INCREMENT value onto ghost table if applicable and user has not specified AUTO_INCREMENT in alter statement

* support GetUint64

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* minor update to test

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* adding test for user defined AUTO_INCREMENT statement

* Generated column as part of UNIQUE (or PRIMARY) KEY

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* skip analysis of generated column data type in unique key

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
2021-05-24 20:16:49 +02:00
Tim Vaillancourt
57a81ceff5
Use GitHub Actions CI badges (#970) 2021-05-19 14:08:32 +02:00
Tim Vaillancourt
c71dbf9ef3
Copy auto increment (#967)
* v1.1.0

* WIP: copying AUTO_INCREMENT value to ghost table
Initial commit: towards setting up a test suite

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* greping for 'expect_table_structure' content

* Adding simple test for 'expect_table_structure' scenario

* adding tests for AUTO_INCREMENT value after row deletes. Should initially fail

* clear event beforehand

* parsing AUTO_INCREMENT from alter query, reading AUTO_INCREMENT from original table, applying AUTO_INCREMENT value onto ghost table if applicable and user has not specified AUTO_INCREMENT in alter statement

* support GetUint64

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* minor update to test

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* adding test for user defined AUTO_INCREMENT statement

Co-authored-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
2021-05-14 15:32:56 +02:00
Tim Vaillancourt
fef83af378
Merge pull request #968 from timvaillancourt/parallel-ci-tests
Use matrix build for replica test CI
2021-05-08 01:11:26 +02:00
Tim Vaillancourt
38bec2e207
Merge branch 'master' into parallel-ci-tests 2021-05-08 01:03:02 +02:00
Tim Vaillancourt
d8bd21acdf
Merge pull request #819 from github/tar-xz-dbdeployer-upgrade
replication tests CI: switch to .tar.xz binaries, upgrade dbdeployer
2021-05-08 01:01:49 +02:00
Tim Vaillancourt
5e37110cfd merge tar-xz-dbdeployer-upgrade 2021-05-08 00:45:59 +02:00
Tim Vaillancourt
a25f63aa2a merge tar-xz-dbdeployer-upgrade 2021-05-08 00:44:28 +02:00
Tim Vaillancourt
cd5beeb495
Remove git branch steps 2021-05-08 00:35:19 +02:00