octoleo
/
lsyncd
mirror of https://github.com/octoleo/lsyncd.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
908 Commits 9 Branches 17 Tags 2.2 MiB
Commit Graph

58 Commits

Author SHA1 Message Date
Daniel Poelzleithner 5096f27bbd Fix more warnings from static code analysis 2022-11-11 15:27:21 +01:00
Daniel Poelzleithner 945b57d8fb Add Cron test 2022-11-11 15:11:09 +01:00
Daniel Poelzleithner 075d64a069 Implement batchSizeLimit for rsync based transfers 
If the batchSizeLimit is set, only files lower then this limit will
be grouped in one rsync transfer.
Each file larger then this limit will spawn their own transfer process.
This will cause large files to no longer block small file transfers under the
circumstance the maxProcess limit on the sync is larger then 1

A very optimized, very secure transfer configuration based on a
pool of ssh connection looks like this:

```
sync {
        default.rsync,
        tunnel = tunnel {
            command = {"ssh", "-N", "-L", "localhost:${localport}:localhost:873", "user@targetmachine"},
            mode = "pool",
            parallel = 2,
        },
        source = "/tmp/src",
        target    = "rsync://localhost:${localport}/test",
        delay = 5,
        batchSizeLimit = 1024 * 1024 * 30,
        maxProcesses = 4,
        rsync = {
                verbose = true,
                inplace = true,
        }
}
```

If you configure remote ssh configuration only allows portforwarding and your rsync daemon
is configured correctly, you can very securely transfer data without giving shell access.
 2022-06-03 06:45:38 +02:00
Daniel Poelzleithner ff84a35d4d Finish full sync cron trigger 2022-06-03 06:45:38 +02:00
Daniel Poelzleithner 46188f9f60 Implement tunnel pool mode. 
In this mode, multiple tunnel processes are started and connection a load
balanced on the pool of connections.

Example config:

...
sync {
  default.rsync,
  tunnel = tunnel {
    command = {"ssh", "-N", "-L", "localhost:${localport}:localhost:873", "user@testmachine"},
    mode = "pool",
    parallel = 2,
  },
  target    = "rsync://localhost:${localport}/test",
  ...
}
 2022-06-03 06:44:02 +02:00
Daniel Poelzleithner 0b59ae7652 Merge remote-tracking branch 'origin/pr/595' 2021-12-10 15:22:23 +01:00
Daniel Poelzleithner f812cd4cd5 Merge remote-tracking branch 'origin/pr/567' 2021-12-10 12:25:17 +01:00
Daniel Poelzleithner 564d54be76 Allow and default to relative paths for binaries executed 2021-12-02 17:51:15 +01:00
Thomas Nixon c4b5465622 allow deleting excluded files with rsync 
This option can't be added to both init and action, because the rsync
exclude mechanism is used to select the files to send, so would delete
all non-changed files. It's only relevant during init anyway, as after
that no excluded files will be made on the destination.
 2020-03-17 00:01:46 +00:00
Dan Rose 390de3dbaa
Expose rsync's copy_unsafe_links 
As per the rsync docs: "This tells rsync to copy the referent of symbolic links that point outside the copied tree. Absolute symlinks are also treated like ordinary files, and so are any symlinks in the source path itself when --relative is used. This option has no additional effect if --copy-links was also specified."
 2019-08-06 18:09:52 -05:00
Axel Kittenberger 465e173983 cleanups 2018-03-01 11:26:12 +01:00
Axel Kittenberger 1e0d867f80 adding inclusion filters 2018-02-27 17:14:36 +01:00
Axel Kittenberger d72a59e70f do not append a '/' to targets when there it ends with a ':' 2018-02-27 10:09:28 +01:00
Axel Kittenberger ea068b4014 reverting to rsync filter useage, until rysnc bug is fixed 2017-02-16 12:17:37 +01:00
Axel Kittenberger 0c5e2c707e new rsync control is no longer recursive, fixing excludes 2017-01-05 11:31:29 +01:00
Axel Kittenberger d1ef5f3d51 use missing-args for rsyncssh too 2017-01-05 10:22:00 +01:00
Axel Kittenberger 0a0fef20ac fixing typos 2017-01-05 10:10:55 +01:00
Axel Kittenberger ccae4ac142 trying delete-missing-args 2017-01-05 10:02:16 +01:00
Axel Kittenberger 4909dd3b2c fixing racecondition in default.rsyncssh 2017-01-03 15:30:13 +01:00
Axel Kittenberger 7bae036f03 lifting some inline funcs 2017-01-03 13:08:48 +01:00
Axel Kittenberger 20edbe5f6e cleanups 2016-12-14 08:45:33 +01:00
Axel Kittenberger 155dbf155c cleanups, removing Lsyncd 2.0.x backward compatibility syntax it was blocked by checkgauge already anyway 2016-12-13 17:40:30 +01:00
Axel Kittenberger 19d0478bd7 added rsync options backup, backup_dir and suffix 2016-12-05 16:38:07 +01:00
Axel Kittenberger db488a86ac adding append/append_verify 2016-12-05 15:47:12 +01:00
Axel Kittenberger 2ff9ae72be adding usermap/groupmap 2016-12-05 15:25:58 +01:00
Axel Kittenberger c58658e642 adding rsync chmod parameter 2016-12-01 14:12:56 +01:00
Axel Kittenberger 319681e7eb added copy_dirlinks 2016-12-01 12:35:02 +01:00
Axel Kittenberger d5957200a2 added --existing 2016-11-28 11:20:12 +01:00
Axel Kittenberger fa3028321b a 2016-08-29 13:14:51 +02:00
Axel Kittenberger 545eb341cf add rsync.chown 2016-08-29 13:12:09 +02:00
Axel Kittenberger 315379072c fixing files with only ] as special char in their name 2016-06-27 09:26:45 +02:00
Marc Abramowitz 86e7a5ab2a Fix typo: Exlcusions => Exclusions 2016-05-23 20:39:10 -07:00
Axel Kittenberger abc2a509f6 omit-[dir/links]-times 2016-05-04 11:25:40 +02:00
Axel Kittenberger 5f74865f68 added inplace option for rsync 2015-10-14 14:39:14 +02:00
Axel Kittenberger 85e95ef150 cosmetics 2014-02-28 10:15:48 +01:00
Axel Kittenberger 6f4613c53a adding ssh.identityFile and ssh.options options 2013-07-30 11:16:29 +02:00
Axel Kittenberger 6290bd6ea7 + fixing 0 characters in -log Exec message for pipes 2013-06-07 11:12:24 +02:00
Axel Kittenberger 25a2274d83 added bwlimit 2013-06-07 11:11:42 +02:00
Kashyap Paidimarri 769fb3c26b Adding timeout to default.rsync 2013-03-19 12:55:34 +05:30
Axel Kittenberger 7bfb535da3 forgot password file in the check_gauge 2012-12-06 10:02:07 +01:00
Axel Kittenberger 78fa6208bf adding a settings('string') get call, so a upstart again operation works again, adding archive to the checkgauge, adding keep_dirs 2012-11-09 20:15:42 +01:00
David Wittman 2c2570e314 Add excludeFrom to default-rsync checkgauge for #164 2012-11-02 18:32:44 -05:00
Axel Kittenberger 3616dc64fe fixing rsync parameter generation, added password-file, thx to foxban 2012-10-30 09:57:54 +01:00
Birger Schmidt e116ce0a6e fix parameter handling for rsync options 2012-10-24 17:46:18 +11:00
Axel Kittenberger d3b31bda37 added temp_dir rsync parameter 2012-10-23 07:23:58 +02:00
Axel Kittenberger 5f01a04335 allow true, false, 'running' and 'startup' for the delete parameter 2012-10-09 17:47:39 +02:00
Axel Kittenberger a6b49c8650 working on the new rsync config system 2012-10-06 14:22:08 +02:00
Axel Kittenberger 6f90c19196 working on new inheritance, chechgauge system 2012-10-06 13:43:55 +02:00
Axel Kittenberger 6a862d6b8f introducing checkgauge, code beautifications 2012-10-05 21:48:06 +02:00
Axel Kittenberger 1bf1d13eaa new inheritance mechanics, code beautifications 2012-10-05 09:41:46 +02:00