octoleo/gh-ost
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
990 Commits 51 Branches 51 Tags 24 MiB
02579075d0
Commit Graph

990 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Shlomi Noach
a4ee80df13 - Building and applying queries from binlog event data! 
- `INSERT`, `DELETE`, `UPDATE` statements
- support for `--noop`
- initial support for `--test-on-replica`. Verifying against `--allow-on-master`
- Changelog events no longer read from binlog stream, because reading it may be throttled, and we have to be able to keep reading the heartbeat and state events.
  They are now being read directly from table, mapping already-seen-events to avoid confusion
  Changlelog listener pools table in 2*frequency of heartbeat injection
 2016-04-14 13:37:56 +02:00
Shlomi Noach
0d25d11b40 added types_test 2016-04-11 19:06:47 +02:00
Shlomi Noach
04525887f3 - Throttling-check is now an async routine running once per second 
- Throttling variables protected by mutex
- Added `--throttle-additional-flag-file`: `operation pauses when this file exists; hint: keep default, use for throttling multiple gh-osc operations`
- ColumnList is not a `struct` which contains ordinal mapping
- More implicit write changelog + audit changelog
- builder now builds `DELETE` and `INSERT` queries from data it will eventually get from DML event
- Sanity check for binlog_row_image
- Restarting replication to be sure binlog settings apply
- Prepare for accepting `SIGHUP` (reloading configuration)
 2016-04-11 17:27:16 +02:00
Shlomi Noach
80163b35b6 extracted on-throttle functions outside loop 2016-04-08 14:44:36 +02:00
Shlomi Noach
a1a34b8150 ongoing development: 
- accepts --max-load
- accepts multiple conditions in --max-load
- throttle includes reason
- chunk-size sanity check
- change log state writes both in appending (history) mode and in replacing (current) mode
- more atomic checks
- inspecting ghost table columns, unique key
- comparing unique keys between tables; sanity
- intersecting columns between tables
- prettify status
- refactored throttle() and retries()
 2016-04-08 14:35:06 +02:00
Shlomi Noach
4652bb7728 Merge pull request #14 from github/ongoing-initial-work 
WIP, ongoing basics
 2016-04-08 10:35:57 +02:00
Shlomi Noach
75f68c0752 - row copy and row events are now handled by a single routine which prioritizes events over rowcopy 
- Supporting `--throttle-file-flag`
- Printing status
- Supporting transactional table syntax
- code cleanup; refactoring
- proper use of atomic where required
- iterations are in changelog (erm... maybe too much)
- `LOCK TABLES`, `UNLOCK TABLES` working
 2016-04-08 10:34:44 +02:00
Shlomi Noach
0e7b23e6fe - Creating an populating Changelog table 
- Using heartbeat
- Throttling works based on heartbeat
- Refactored binlog_reader stuff. Now streaming events (into golang channel, which makes for nice buffering and throttling)
- Binlog table listeners work
- More Migrator logic; existing logic for waiting on `state` events (e.g. `TablesCreatedState`)
 2016-04-07 15:57:12 +02:00
Shlomi Noach
4dd5a93ed7 - More work on go-mysql reader: now properly notes binlog coordinates as it makes progress 
- Properly creating multiple BinlogEntry-ies when reading multi statement binlog event (Still todo: transactions)
 2016-04-06 18:44:54 +02:00
Shlomi Noach
5deff2adb6 - working POC of row-copy iteration cycle 
- initial work on table columns
- initial work on events streamer
 2016-04-06 13:05:58 +02:00
Shlomi Noach
f771016bd5 . 2016-04-06 13:05:21 +02:00
Shlomi Noach
d8fefb3d6f exploded args on range query building; iteration works 2016-04-05 19:50:49 +02:00
Shlomi Noach
3583ab5dc5 beginning support for ranges and iteration. Still WIP 2016-04-05 09:14:22 +02:00
Shlomi Noach
ea0906f4e5 reading table (range) min/max values, right now according to hardcoded unique key 2016-04-04 18:19:46 +02:00
Shlomi Noach
937491674c adding applier, instance_key, instance_key_map 2016-04-04 15:30:49 +02:00
Shlomi Noach
cf87d16044 detecting master (includes sanity checks). Introducing Applier. Creating and altering ghost table 2016-04-04 15:29:02 +02:00
Shlomi Noach
bba352922a Merge pull request #10 from github/initial-db-access 
Initial db access
 2016-04-04 12:29:35 +02:00
Shlomi Noach
c75cd998fb a bunch of 'inspector' initial tests on the replica 2016-04-04 12:27:51 +02:00
Shlomi Noach
1b8c5ff286 preparing for db access 2016-04-01 16:05:54 +02:00
Shlomi Noach
f5b276415a initial work on context 2016-04-01 16:05:44 +02:00
Shlomi Noach
b461c320cc added BuildUniqueKeyRangeEndPreparedQuery() 2016-04-01 16:05:17 +02:00
Shlomi Noach
f0e37cde1c added vendor packages 2016-04-01 16:04:27 +02:00
Shlomi Noach
5e5b868c5c Merge pull request #9 from github/build-range-prepared 
building prepared queries (with positional params)
 2016-04-01 13:56:33 +02:00
Shlomi Noach
41c3d15963 building prepared queries (with positional params) 2016-04-01 13:55:51 +02:00
Shlomi Noach
9b368a7720 Merge pull request #8 from github/sql-queries-manipulations 
merging so I can use this on other branches
 2016-04-01 13:54:20 +02:00
Shlomi Noach
20a74d578d Merge pull request #6 from github/experimenting-go-mysql 
merging so I can use this on other branches
 2016-04-01 13:54:07 +02:00
Shlomi Noach
39ebc75c43 initial work on sql query building 2016-04-01 13:36:56 +02:00
Shlomi Noach
8f3d13e071 still experimenting, yes? go-mysql binlog parser looks good 2016-03-30 15:43:40 +02:00
Shlomi Noach
e3210a9fa2 Merge pull request #4 from github/parsing-mysqlbinlog 
further work on mysqlbinlog_reader
 2016-03-30 10:25:59 +02:00
Shlomi Noach
30a472f741 begin work on positional columns. Incomplete 2016-03-24 22:27:39 +01:00
Shlomi Noach
96a8fd50c3 minor refactoring; fixed license comments 2016-03-24 15:11:56 +01:00
Shlomi Noach
fb04eb232f simplified appendBinlogEntry() 2016-03-24 14:52:49 +01:00
Shlomi Noach
8aa6a9750c adjusted automaton to accept multi-statement entries; added tests 2016-03-24 14:25:52 +01:00
Shlomi Noach
1768b55b3b mysqlbinlog_reader is now a simple state machine 2016-03-23 15:25:45 +01:00
Shlomi Noach
e088a3abac Merge pull request #2 from github/initial-code 
adding initial bunch of code
 2016-03-23 13:53:30 +01:00
Shlomi Noach
e36bb51b69 initial seems-to-be-working parser for binary logs, which reads log pos, end log pos, statement type, schema and table. Performs some validations along the way. Reads in chunks of up to 32M per chunk 2016-03-23 12:40:17 +01:00
Shlomi Noach
1576119576 adding build file 2016-03-23 12:39:24 +01:00
Shlomi Noach
09046c8f6e initial binlog package 2016-03-22 15:12:51 +01:00
Shlomi Noach
bb1c6dae30 adding initial bunch of code 2016-03-21 15:57:01 +01:00
Shlomi Noach
295d98c5b2 Initial commit 2016-03-21 11:08:52 +01:00