octoleo
/
lsyncd
mirror of https://github.com/octoleo/lsyncd.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
871 Commits 9 Branches 17 Tags 2.2 MiB
Commit Graph

353 Commits

Author SHA1 Message Date
Daniel Poelzleithner 075d64a069 Implement batchSizeLimit for rsync based transfers 
If the batchSizeLimit is set, only files lower then this limit will
be grouped in one rsync transfer.
Each file larger then this limit will spawn their own transfer process.
This will cause large files to no longer block small file transfers under the
circumstance the maxProcess limit on the sync is larger then 1

A very optimized, very secure transfer configuration based on a
pool of ssh connection looks like this:

```
sync {
        default.rsync,
        tunnel = tunnel {
            command = {"ssh", "-N", "-L", "localhost:${localport}:localhost:873", "user@targetmachine"},
            mode = "pool",
            parallel = 2,
        },
        source = "/tmp/src",
        target    = "rsync://localhost:${localport}/test",
        delay = 5,
        batchSizeLimit = 1024 * 1024 * 30,
        maxProcesses = 4,
        rsync = {
                verbose = true,
                inplace = true,
        }
}
```

If you configure remote ssh configuration only allows portforwarding and your rsync daemon
is configured correctly, you can very securely transfer data without giving shell access.
 2022-06-03 06:45:38 +02:00
Daniel Poelzleithner 4a0896f58f fix logic error in alarm compare 2022-06-03 06:45:38 +02:00
Daniel Poelzleithner 6d9164a29a Fix wrong alarm selection 2022-06-03 06:45:38 +02:00
Daniel Poelzleithner ff84a35d4d Finish full sync cron trigger 2022-06-03 06:45:38 +02:00
Daniel Poelzleithner 9ec4f42888 Only check Tunnels if there is at least one tunnel defined 2022-06-03 06:45:38 +02:00
Daniel Poelzleithner 95f8ea67dd Log source of next alarm as well 2022-06-03 06:45:38 +02:00
Daniel Poelzleithner c8fbe955fe Add debug functions and timestamp2string helpers 2022-06-03 06:45:38 +02:00
Daniel Poelzleithner 58f1991e0b [WIP] implement crontab full sync 2022-06-03 06:45:38 +02:00
Daniel Poelzleithner 46188f9f60 Implement tunnel pool mode. 
In this mode, multiple tunnel processes are started and connection a load
balanced on the pool of connections.

Example config:

...
sync {
  default.rsync,
  tunnel = tunnel {
    command = {"ssh", "-N", "-L", "localhost:${localport}:localhost:873", "user@testmachine"},
    mode = "pool",
    parallel = 2,
  },
  target    = "rsync://localhost:${localport}/test",
  ...
}
 2022-06-03 06:44:02 +02:00
Daniel Poelzleithner 6e60f6b89e Call runner cleanup method when exiting through error codes 2022-06-03 06:44:02 +02:00
Daniel Poelzleithner 2b9de7d4c3 rename traceback function 2022-06-03 06:44:02 +02:00
Daniel Poelzleithner fb86124f30 Call user provideable check function 2022-06-03 06:44:01 +02:00
Daniel Poelzleithner f66b7147fd Report tunnel status in status file 2022-06-03 06:44:01 +02:00
Daniel Poelzleithner c543a21162 Add function to substitude commands with placeholders 2022-06-03 06:44:01 +02:00
Daniel Poelzleithner 3d2288dccf Updates to tunnel logic. 
Delay before tunnel is considered up.
Working delay if restart fails.
Disabled state.
Kill all tunnel processes on graceful exit
 2022-06-03 06:44:01 +02:00
Daniel Poelzleithner f603d41c6c bump version 2022-06-03 06:44:01 +02:00
Daniel Poelzleithner 274b2b0416 [WIP] Add tunnel support 2022-06-03 06:44:01 +02:00
Daniel Poelzleithner e2a27af6e7 use relative sh path 2022-01-12 20:46:41 +01:00
Daniel Poelzleithner 39f2f3a373 Finish -onepass option to exit when all syncs ran sucessfully once 2021-12-10 15:05:30 +01:00
Daniel Poelzleithner d737c03c03 Merge remote-tracking branch 'origin/pr/541' 2021-12-10 12:51:01 +01:00
Daniel Poelzleithner 3e8aad3b2e Use sshopts for rsyncssh tests 2021-12-08 18:26:11 +01:00
Daniel Poelzleithner 5b0d266669 add cmd option to pass ssh options 2021-12-08 18:24:13 +01:00
Bernard Gray a57b80f9c8 make indenting consistent 2018-12-06 08:41:53 +11:00
Bernard Gray 188b691bea add onepass option to exit after initial sync 2018-12-05 17:05:18 +11:00
Axel Kittenberger 99b6b18ac2 fixing loadstring() Lua5.3 issue 2018-03-09 10:42:10 +01:00
Axel Kittenberger 4bc456b544 preparing 2.2.3 2018-03-09 10:13:29 +01:00
Axel Kittenberger faa582e259 cleanups 2018-03-01 15:19:30 +01:00
Axel Kittenberger ba52ee1a6f cleanups 2018-03-01 15:08:26 +01:00
Axel Kittenberger a78f239fa2 apply filters before inotify watching dirs, actually filter events 2018-03-01 14:14:28 +01:00
Axel Kittenberger 465e173983 cleanups 2018-03-01 11:26:12 +01:00
Axel Kittenberger 1e0d867f80 adding inclusion filters 2018-02-27 17:14:36 +01:00
Axel Kittenberger d72a59e70f do not append a '/' to targets when there it ends with a ':' 2018-02-27 10:09:28 +01:00
Axel Kittenberger ea068b4014 reverting to rsync filter useage, until rysnc bug is fixed 2017-02-16 12:17:37 +01:00
Axel Kittenberger 785a7dd503 fixing again setting with new delay objects, cleanups 2017-02-06 17:00:39 +01:00
Axel Kittenberger 544f6066b7 write pid of forked process 2017-01-09 13:13:05 +01:00
Axel Kittenberger 3f78514273 fix exclusion tests 2017-01-09 11:43:23 +01:00
Axel Kittenberger 288f09ad01 a 2017-01-09 11:14:23 +01:00
Axel Kittenberger e21a2184ba preparing 2.2.1 2017-01-05 11:54:29 +01:00
Axel Kittenberger 0c5e2c707e new rsync control is no longer recursive, fixing excludes 2017-01-05 11:31:29 +01:00
Axel Kittenberger b7c4fe437c typo 2017-01-05 10:29:38 +01:00
Axel Kittenberger ccae4ac142 trying delete-missing-args 2017-01-05 10:02:16 +01:00
Axel Kittenberger 12d4f43d0f realtive paths do not start with a slash 2017-01-04 11:24:55 +01:00
Axel Kittenberger c08719ce00 Preparing 2.2.0, removing manpage from 'clean' target 2017-01-04 10:59:01 +01:00
Axel Kittenberger 4909dd3b2c fixing racecondition in default.rsyncssh 2017-01-03 15:30:13 +01:00
Axel Kittenberger 83c6436e84 typo 2016-12-22 15:15:17 +01:00
Axel Kittenberger 16919a893e do not replace active delays 2016-12-22 14:59:18 +01:00
Axel Kittenberger 62923459f4 Actually stack when toDelete,stack 2016-12-22 13:55:59 +01:00
Axel Kittenberger 98c83a7fbf typo 2016-12-21 16:38:04 +01:00
Axel Kittenberger 8e1a85a12d Splitted Delay combination log from logging about it. 2016-12-21 16:29:29 +01:00
Axel Kittenberger 837a3cd927 Fixed Delay changes potentially not reflect on Event layer. Fixed test exitcode 2016-12-21 13:11:55 +01:00