Update docs

This commit is contained in:
Daniel Poelzleithner 2022-06-03 04:24:51 +02:00
parent f244405bce
commit e94edac51e
2 changed files with 174 additions and 55 deletions

View File

@ -15,7 +15,7 @@ sync {
}
{% endhighlight %}
The order of the arguments is of no importance. If target is a local directory, take care that it is an absolute pathname. You can add multiple syncs that way. The source directories may be identical or differ without problems. ```source``` is an universal parameter that must be given for every sync. All other ```sync``` parameters can differ depending on the behavior selected. Optionally you can override the default or settings values ```maxDelays``` or ```maxProcesses``` per _Sync_.
The order of the arguments is of no importance. If target is a local directory, take care that it is an absolute pathname. You can add multiple syncs that way. The source directories may be identical or differ without problems. ```source``` is an universal parameter that must be given for every sync. All other ```sync``` parameters can differ depending on the behavior selected. Optionally you can override the default or settings values ```maxDelays``` or ```maxProcesses``` per _Sync_.
One can also skip the initial rsync process by setting the default ```init``` function to false:
@ -32,8 +32,20 @@ This is an optimization which can be dangerous; so, please use it only if you ar
The default behaviors you can select from are following:
Shared Settings
---------------
Following settings are shared between all defaults:
| Name | Description |
|-----------------|------------------|
| source | Source directory |
default.rsync
-------------
The default rsync configuration will aggregate events up to ```delay``` seconds or 1000 separate uncollapsible events, which ever happens first. Then it will spawn one Rsync with a filter of all files that changed. The filter list is transmitted to Rsync trough a pipe. A call from Lsyncd to Rsync will thus look like this:
{% highlight shell %}
@ -49,7 +61,7 @@ sync {
default.rsync,
source = "/home/user/src/",
target = "foohost.com:~/trg/",
delay = 15,
delay = 15,
rsync = {
binary = "/usr/local/bin/rsync",
archive = true,
@ -58,6 +70,12 @@ sync {
}
{% endhighlight %}
Additional settings:
| Name | Description |
|-----------------|-------------|
| batchSizeLimit | Files larger then this limit should not be batched into on transfer. Only makes sense with processes > 1 which prevents rsyncssh |
Below is a table of options for the ```rsync``` parameter. Please have a look at the Rsync documentation for an in depth explanation.
<table>
@ -67,7 +85,7 @@ Below is a table of options for the ```rsync``` parameter. Please have a look at
</td><td> <b>default value</b>
</td><td> <b>comment</b>
</td></tr>
<tr><td> acls
</td><td> =
</td><td> BOOL
@ -127,14 +145,14 @@ Below is a table of options for the ```rsync``` parameter. Please have a look at
<tr><td> chmod
</td><td> =
</td><td> STRING
</td><td>
</td><td>
</td><td> (Lsyncd >= 2.2.0)
</td></tr>
<tr><td> chown
</td><td> =
</td><td> USER:GROUP
</td><td>
</td><td>
</td><td> (Lsyncd >= 2.2.0)
</td></tr>
@ -175,7 +193,7 @@ Below is a table of options for the ```rsync``` parameter. Please have a look at
<tr><td> exclude
</td><td> =
</td><td> PATTERN
</td><td>
</td><td>
</td><td> TABLE of PATTERNs also allowed
</td></tr>
@ -183,7 +201,7 @@ Below is a table of options for the ```rsync``` parameter. Please have a look at
</td><td> =
</td><td> FILENAME
</td><td>
</td><td>
</td><td>
</td></tr>
<tr><td> executability
@ -210,7 +228,7 @@ Below is a table of options for the ```rsync``` parameter. Please have a look at
<tr><td> groupmap
</td><td> =
</td><td> STRING
</td><td>
</td><td>
</td><td> (Lsyncd >= 2.2.0)
</td></tr>
@ -239,21 +257,21 @@ Below is a table of options for the ```rsync``` parameter. Please have a look at
</td><td> =
</td><td> BOOL
</td><td> false
</td><td>
</td><td>
</td></tr>
<tr><td> ipv6
</td><td> =
</td><td> BOOL
</td><td> false
</td><td>
</td><td>
</td></tr>
<tr><td> links
</td><td> =
</td><td> BOOL
</td><td> true
</td><td>
</td><td>
</td></tr>
<tr><td> one_file_system
@ -273,7 +291,7 @@ Below is a table of options for the ```rsync``` parameter. Please have a look at
<tr><td> password_file
</td><td> =
</td><td> FILENAME
</td><td>
</td><td>
</td><td> (Lsyncd >= 2.1.2)
</td></tr>
@ -308,14 +326,14 @@ Below is a table of options for the ```rsync``` parameter. Please have a look at
<tr><td> rsh
</td><td> =
</td><td> COMMAND
</td><td>
</td><td>
</td><td>
</td><td>
</td></tr>
<tr><td> rsync_path
</td><td> =
</td><td> PATH
</td><td>
</td><td>
</td><td> (path to rsync on remote host)
</td></tr>
@ -329,14 +347,14 @@ Below is a table of options for the ```rsync``` parameter. Please have a look at
<tr><td> suffix
</td><td> =
</td><td> SUFFIX
</td><td>
</td><td>
</td><td> (Lsyncd >= 2.2.0)
</td></tr>
<tr><td> temp_dir
</td><td> =
</td><td> DIR
</td><td>
</td><td>
</td><td>
</td></tr>
@ -344,7 +362,7 @@ Below is a table of options for the ```rsync``` parameter. Please have a look at
</td><td> =
</td><td> BOOL
</td><td> true
</td><td>
</td><td>
</td></tr>
<tr><td> update
@ -385,7 +403,7 @@ Below is a table of options for the ```rsync``` parameter. Please have a look at
<tr><td> _extra
</td><td> =
</td><td> TABLE of STRINGS.
</td><td>
</td><td>
</td><td> If absolutely needed, additional arguments can be specified as a TABLE of STRINGS(example: <tt>{ "--omit-dir-times", "--omit-link-times" }</tt>). Note that the underscore highlights this as workaround. If you need something that is not covered by the above options, please request it via a feature request on the project website. Most notably, do not add -r for recursive or -a which implies recursive, since Lsyncd will handle that by itself. Additionally do not add -R for relative, which will ruin Lsyncd &lt;-&gt; Rsync communication.
</td></tr>
@ -393,7 +411,8 @@ Below is a table of options for the ```rsync``` parameter. Please have a look at
default.rsyncssh
----------------
This configuration differs from the standard rsync configuration in that it uses ssh commands to move files or directories locally at the target host instead of deleting and transferring again. This configuration does spawn Rsync processes like default.rsync but additionally will spawn ```/usr/bin/ssh HOST mv ORIGIN DESTINATION``` commands.
This configuration differs from the standard rsync configuration in that it uses ssh commands to move files or directories locally at the target host instead of deleting and transferring again. This configuration does spawn Rsync processes like default.rsync but additionally will spawn ```/usr/bin/ssh HOST mv ORIGIN DESTINATION``` commands.
Different to default.rsync it does not take an uniform ```target``` parameter, but needs ```host``` and ```targetdir``` separated.
@ -463,8 +482,8 @@ sync {
Please note the comma between the ```rsync``` parameter set and the ```ssh``` parameter set.
**Caution**
If you are upgrading from 2.0.x, please notice that `settings` became a function from a variable, so you **MUST** delete the equal sign '=' between `settings` and the `{`.
__Caution__
If you are upgrading from 2.0.x, please notice that `settings` became a function from a variable, so you __MUST__ delete the equal sign '=' between `settings` and the `{`.
Lsyncd will call ```xargs``` on the remote host to handle multiple tasks in a single connection. Xargs options can be specified by the xargs parameter.
@ -490,7 +509,6 @@ Lsyncd will call ```xargs``` on the remote host to handle multiple tasks in a si
</table>
Example:
{% highlight lua %}
@ -504,6 +522,7 @@ sync {
default.direct
-------------
Default.direct can be used to keep two local directories in sync with better performance than using default.rsync. Default.direct uses (just like default.rsync) rsync on startup to initially synchronize the target directory with the source directory. However, during normal operation default.direct uses /bin/cp, /bin/rm and /bin/mv to keep the synchronization. All parameters are just like default.rsync.
Example:
@ -518,6 +537,7 @@ sync {
Exclusions
----------
Two additional parameters can be specified to sync{}:
<table>
@ -538,12 +558,12 @@ Two additional parameters can be specified to sync{}:
Exclusion rules are modeled after rsync's exclusion patterns but are a bit simpler. Lsyncd supports these features:
* Generally if any segment of the pathname (see below Layer 3) of an event matches the text, it is excluded. E.g. the file "/bin/foo/bar" matches the rule "foo".
* If the rule starts with a slash, it will only be matched at the beginning of the pathname
* If the rule ends with a slash, it will only be matched at the end of a pathname
* ? matches any character that is not a slash.
* ```*``` matches zero or more characters that are not a slash
* ```**``` matches zero or more characters, this can be slashes.
* Generally if any segment of the pathname (see below Layer 3) of an event matches the text, it is excluded. E.g. the file "/bin/foo/bar" matches the rule "foo".
* If the rule starts with a slash, it will only be matched at the beginning of the pathname
* If the rule ends with a slash, it will only be matched at the end of a pathname
* ? matches any character that is not a slash.
* ```*``` matches zero or more characters that are not a slash
* ```**``` matches zero or more characters, this can be slashes.
Example:
@ -552,13 +572,14 @@ sync {
default.rsync,
source = "/home/user/src/",
targetdir = "/home/user/dst/",
exclude = { '*.bak' , '*.tmp' }
exclude = { '_.bak' , '_.tmp' }
}
{% endhighlight %}
Deletions
---------
By default Lsyncd will delete files on the target that are not present at the source since this is a fundamental part of the idea of keeping the target in sync with the source. However, many users requested exceptions for this, for various reasons, so all default implementations take ```delete``` as an additional parameter.
By default Lsyncd will delete files on the target that are not present at the source since this is a fundamental part of the idea of keeping the target in sync with the source. However, many users requested exceptions for this, for various reasons, so all default implementations take ```delete``` as an additional parameter.
Valid values for ```delete``` are:
@ -589,3 +610,87 @@ Valid values for ```delete``` are:
</td></tr>
</table>
Tunnels
-------
*New in: 2.3.0*
Lsyncd is able to start and manage external programs to provide a tunnel for data transfer.
Additionally it can spawn multiple connections and load-balance connections among them. A tunnel is created through the `tunnel` function.
```
{% highlight lua %}
sync {
default.rsync,
tunnel = tunnel {
command = {"ssh", "-N", "-L", "localhost:5432:localhost:873", "tunnel@testmachine"},
}
target = "rsync://localhost:5432/projects",
source = "/home/user/src/",
}
{% endhighlight %}
```
You can then set the shell for the tunnel user to `/bin/false` and configure the rsyncd server side appropriately.
Valid arguments for `tunnel` are:
| Argument | Description | Default | Valid values |
|------------|-------------------------------------------------------|-----------|-------------------------------|
| mode | Mode in which the tunnel is run | command | command, pool |
| command | _Required_ Command to run | nil | Table of arguments |
| parallel | How many connections to run. Only for pool mode | 1 | 1+ |
| retryDelay | Seconds to wait until tunnel is restarted | 10 | Number |
| readyDelay | Seconds after program start to consider tunnel up | 5 | Number |
| localhost | Name of the local host variable | localhost | String |
## Pool Mode
In pool mode, lsyncd allocates a new local port which is then passed as variable to the host command.
All variables can be substituded by `^variable` syntax.
See [../../../../examples/lrsyncssh-tunnel.lua](lrsyncssh-tunnel.lua) for a extended configuration.
### List of variables
| Name | Description |
|------------|-----------------------------------------------|
| localport | Port allocated for the selected connection |
| localhost | Local hostname used. Default localhost |
### Example
Run 2 tunnel ssh processes and 4 rsync processes at the same time. Use extra transfers for all files larger 30 MB.
```lua
sync {
default.rsync,
tunnel = tunnel {
command = {"ssh", "-N", "-L", "localhost:^localport:localhost:873", "user@testmachine"},
mode = "pool",
parallel = 2,
},
source = "/data/projects",
target = "rsync://localhost:^localport/projects",
delay = 5,
batchSizeLimit = 1024 * 1024 * 30,
maxProcesses = 4
}
```
This example will open 2 ssh connections for port forwarding and load balance 4 parallel rsync
processes in a round roubin fashion.
### Workflow in Poolmodel
When a sync with a tunnel parameter is started, all events are queued until the tunnel reaches the
`UP` state, which is when one successful tunnel process exists for at least `readyDelay` seconds.
Dead tunnel processes are automatically restarted. When the tunnel process count drops to 0, tunnel falls back to the
`CONNECTING` state. There is a `retryDelay` seconds delay between each attempt to restart the tunnel.
Once the tunnel is UP, a full transfer is initiated. Subsequent transfers are then load balanced over multiple connections.
### Notes on Poolmode
* Pool mode only works with `rsync` backend, since there is no way to prevent multiple transfers to the same file in a relieable way, the rsync backend only supports `maxProcesses = 1` which renders pool mode useless. Since the remote side rsync daemon can prevent file trashing, the rsync backend is safe.

View File

@ -4,73 +4,87 @@ title: Invoking
---
As most Unix tools, Lsyncd will print a synopsis of its command line options when called with --help.
{% highlight shell %}
```console
lsyncd --help
lsyncd -help
{% endhighlight %}
```
The two hyphens are redundant for Lsyncd. It has no short one letter options and one hyphen will always result into the same as specifying two.
Also like most Unix tools, ```--version``` or ```-version``` will let Lsyncd print its version number.
{% highlight shell %}
```console
lsyncd -version
{% endhighlight %}
```
Lsyncd 2.1 is designed to be predominantly configured through a config file (see below). The config file can thus be the only command line option.
{% highlight shell %}
```console
lsyncd CONFIGFILE
{% endhighlight %}
```
Although for standard use or quick testing it can be cursorily configured by command line options. The following will keep a local source and destination directory in sync using rsync:
{% highlight shell %}
```console
lsyncd -rsync /home/USER/src /home/USER/dst
{% endhighlight %}
```
The target can here be anything that Rsync recognizes.
{% highlight shell %}
```console
lsyncd -rsync /home/USER/src remotehost:dst
{% endhighlight %}
```
Two (or more) targets are configured by calling -rsync twice (or several times).
{% highlight shell %}
```console
lsyncd -rsync /home/USER/src remotehost1:dst -rsync /home/USER/src remotehost2:dst
{% endhighlight %}
```
A disadvantage with Rsync synchronization is that normally directory and file moves result in a deletion of the move origin and a retransfer of the move destination of the wire. However, Lsyncd 2 can use ssh commands to move the directories and files locally on the target. To use this use ```-rsyncssh``` followed by the local source directory, the remote host and the target directory there. The REMOTEHOST can include a user like ```me@remotehost.com```.
{% highlight shell %}
```console
lsyncd -rsyncssh /home/USER/src REMOTEHOST TARGETDIR
{% endhighlight %}
```
When testing Lsyncd configurations ```-nodaemon``` is a pretty handy flag. With this option, Lsyncd will not detach and will not become a daemon. All log messages are additionally to the configured logging facilities printed on the console (_stdout_ and _stderr_).
{% highlight shell %}
lsyncd -nodaemon CONFIGFILE
{% endhighlight %}
There is a difference in behaviour when running with -nodaemon. Lsyncd will not change its working directory to ```/```, as it does when becoming a daemon. Thus relative targets like ```./target``` will work with ```-nodaemon``` but must be specified to absolute paths to work in daemon mode. The source directories will also be turned into absolute paths by Lsyncd. The reason targets are not resolved to absolute paths while sources are is because Lsyncd itself does not care about the format of the target specifier which can also be remote hosts, rsyncd modules, etc. It is opaquely handed to rsync. It cares about the observed directories though.
```console
lsyncd -nodaemon CONFIGFILE
```
There is a difference in behaviour when running with -nodaemon. Lsyncd will not change its working directory to `/`, as it does when becoming a daemon. Thus relative targets like `./target` will work with `-nodaemon` but must be specified to absolute paths to work in daemon mode. The source directories will also be turned into absolute paths by Lsyncd. The reason targets are not resolved to absolute paths while sources are is because Lsyncd itself does not care about the format of the target specifier which can also be remote hosts, rsyncd modules, etc. It is opaquely handed to rsync. It cares about the observed directories though.
*New in 2.3.0*
To only trigger the initial full sync and not monitor the file system, you can use the `-onepass` option.
```console
lsyncd -onepass CONFIGFILE
```
## Logging
All log messages are sorted in categories. By default Lsyncd is scarce with log messages. You can turn Lsyncd into a motormouth by specifying ```-log all```.
{% highlight shell %}
```console
lsyncd -log all CONFIGFILE
{% endhighlight %}
```
This might easily become too much. A particularly useful category is "Exec" which will log the command lines of all processes Lsyncd spawns.
{% highlight shell %}
```console
lsyncd -log Exec CONFIGFILE
{% endhighlight %}
```
When the initial startup sync fails by default Lsyncd will terminate with an error message. It has been designed this way, so configuration failures are visibly reported to a possibly beginning user. However, in production a remote target might be done, but you want Lsyncd to start nevertheless and keep trying to sync to the remote target until it is up.
{% highlight shell %}
```console
lsyncd -insist -rsync /home/USER/src remotehost:dst
{% endhighlight %}
```
In production mode it is recommended to have insist on. It can also be specified in the settings{} command in a config file.