Commit Graph

56 Commits

Author SHA1 Message Date
Daniel Poelzleithner
075d64a069 Implement batchSizeLimit for rsync based transfers
If the batchSizeLimit is set, only files lower then this limit will
be grouped in one rsync transfer.
Each file larger then this limit will spawn their own transfer process.
This will cause large files to no longer block small file transfers under the
circumstance the maxProcess limit on the sync is larger then 1

A very optimized, very secure transfer configuration based on a
pool of ssh connection looks like this:

```
sync {
        default.rsync,
        tunnel = tunnel {
            command = {"ssh", "-N", "-L", "localhost:${localport}:localhost:873", "user@targetmachine"},
            mode = "pool",
            parallel = 2,
        },
        source = "/tmp/src",
        target    = "rsync://localhost:${localport}/test",
        delay = 5,
        batchSizeLimit = 1024 * 1024 * 30,
        maxProcesses = 4,
        rsync = {
                verbose = true,
                inplace = true,
        }
}
```

If you configure remote ssh configuration only allows portforwarding and your rsync daemon
is configured correctly, you can very securely transfer data without giving shell access.
2022-06-03 06:45:38 +02:00
Daniel Poelzleithner
ff84a35d4d Finish full sync cron trigger 2022-06-03 06:45:38 +02:00
Daniel Poelzleithner
46188f9f60 Implement tunnel pool mode.
In this mode, multiple tunnel processes are started and connection a load
balanced on the pool of connections.

Example config:

...
sync {
  default.rsync,
  tunnel = tunnel {
    command = {"ssh", "-N", "-L", "localhost:${localport}:localhost:873", "user@testmachine"},
    mode = "pool",
    parallel = 2,
  },
  target    = "rsync://localhost:${localport}/test",
  ...
}
2022-06-03 06:44:02 +02:00
Daniel Poelzleithner
0b59ae7652 Merge remote-tracking branch 'origin/pr/595' 2021-12-10 15:22:23 +01:00
Daniel Poelzleithner
f812cd4cd5 Merge remote-tracking branch 'origin/pr/567' 2021-12-10 12:25:17 +01:00
Daniel Poelzleithner
564d54be76 Allow and default to relative paths for binaries executed 2021-12-02 17:51:15 +01:00
Thomas Nixon
c4b5465622 allow deleting excluded files with rsync
This option can't be added to both init and action, because the rsync
exclude mechanism is used to select the files to send, so would delete
all non-changed files. It's only relevant during init anyway, as after
that no excluded files will be made on the destination.
2020-03-17 00:01:46 +00:00
Dan Rose
390de3dbaa
Expose rsync's copy_unsafe_links
As per the rsync docs: "This tells rsync to copy the referent of symbolic links that point outside the copied tree. Absolute symlinks are also treated like ordinary files, and so are any symlinks in the source path itself when --relative is used. This option has no additional effect if --copy-links was also specified."
2019-08-06 18:09:52 -05:00
Axel Kittenberger
465e173983 cleanups 2018-03-01 11:26:12 +01:00
Axel Kittenberger
1e0d867f80 adding inclusion filters 2018-02-27 17:14:36 +01:00
Axel Kittenberger
d72a59e70f do not append a '/' to targets when there it ends with a ':' 2018-02-27 10:09:28 +01:00
Axel Kittenberger
ea068b4014 reverting to rsync filter useage, until rysnc bug is fixed 2017-02-16 12:17:37 +01:00
Axel Kittenberger
0c5e2c707e new rsync control is no longer recursive, fixing excludes 2017-01-05 11:31:29 +01:00
Axel Kittenberger
d1ef5f3d51 use missing-args for rsyncssh too 2017-01-05 10:22:00 +01:00
Axel Kittenberger
0a0fef20ac fixing typos 2017-01-05 10:10:55 +01:00
Axel Kittenberger
ccae4ac142 trying delete-missing-args 2017-01-05 10:02:16 +01:00
Axel Kittenberger
4909dd3b2c fixing racecondition in default.rsyncssh 2017-01-03 15:30:13 +01:00
Axel Kittenberger
7bae036f03 lifting some inline funcs 2017-01-03 13:08:48 +01:00
Axel Kittenberger
20edbe5f6e cleanups 2016-12-14 08:45:33 +01:00
Axel Kittenberger
155dbf155c cleanups, removing Lsyncd 2.0.x backward compatibility syntax it was blocked by checkgauge already anyway 2016-12-13 17:40:30 +01:00
Axel Kittenberger
19d0478bd7 added rsync options backup, backup_dir and suffix 2016-12-05 16:38:07 +01:00
Axel Kittenberger
db488a86ac adding append/append_verify 2016-12-05 15:47:12 +01:00
Axel Kittenberger
2ff9ae72be adding usermap/groupmap 2016-12-05 15:25:58 +01:00
Axel Kittenberger
c58658e642 adding rsync chmod parameter 2016-12-01 14:12:56 +01:00
Axel Kittenberger
319681e7eb added copy_dirlinks 2016-12-01 12:35:02 +01:00
Axel Kittenberger
d5957200a2 added --existing 2016-11-28 11:20:12 +01:00
Axel Kittenberger
fa3028321b a 2016-08-29 13:14:51 +02:00
Axel Kittenberger
545eb341cf add rsync.chown 2016-08-29 13:12:09 +02:00
Axel Kittenberger
315379072c fixing files with only ] as special char in their name 2016-06-27 09:26:45 +02:00
Marc Abramowitz
86e7a5ab2a Fix typo: Exlcusions => Exclusions 2016-05-23 20:39:10 -07:00
Axel Kittenberger
abc2a509f6 omit-[dir/links]-times 2016-05-04 11:25:40 +02:00
Axel Kittenberger
5f74865f68 added inplace option for rsync 2015-10-14 14:39:14 +02:00
Axel Kittenberger
85e95ef150 cosmetics 2014-02-28 10:15:48 +01:00
Axel Kittenberger
6f4613c53a adding ssh.identityFile and ssh.options options 2013-07-30 11:16:29 +02:00
Axel Kittenberger
6290bd6ea7 + fixing 0 characters in -log Exec message for pipes 2013-06-07 11:12:24 +02:00
Axel Kittenberger
25a2274d83 added bwlimit 2013-06-07 11:11:42 +02:00
Kashyap Paidimarri
769fb3c26b Adding timeout to default.rsync 2013-03-19 12:55:34 +05:30
Axel Kittenberger
7bfb535da3 forgot password file in the check_gauge 2012-12-06 10:02:07 +01:00
Axel Kittenberger
78fa6208bf adding a settings('string') get call, so a upstart again operation works again, adding archive to the checkgauge, adding keep_dirs 2012-11-09 20:15:42 +01:00
David Wittman
2c2570e314 Add excludeFrom to default-rsync checkgauge for #164 2012-11-02 18:32:44 -05:00
Axel Kittenberger
3616dc64fe fixing rsync parameter generation, added password-file, thx to foxban 2012-10-30 09:57:54 +01:00
Birger Schmidt
e116ce0a6e fix parameter handling for rsync options 2012-10-24 17:46:18 +11:00
Axel Kittenberger
d3b31bda37 added temp_dir rsync parameter 2012-10-23 07:23:58 +02:00
Axel Kittenberger
5f01a04335 allow true, false, 'running' and 'startup' for the delete parameter 2012-10-09 17:47:39 +02:00
Axel Kittenberger
a6b49c8650 working on the new rsync config system 2012-10-06 14:22:08 +02:00
Axel Kittenberger
6f90c19196 working on new inheritance, chechgauge system 2012-10-06 13:43:55 +02:00
Axel Kittenberger
6a862d6b8f introducing checkgauge, code beautifications 2012-10-05 21:48:06 +02:00
Axel Kittenberger
1bf1d13eaa new inheritance mechanics, code beautifications 2012-10-05 09:41:46 +02:00
Axel Kittenberger
2e9c103f55 beautifing code 2012-10-03 18:34:09 +02:00
Axel Kittenberger
e2b0cdb359 fixing new rsync argument computation 2012-10-02 08:25:46 +02:00