mirror of
https://github.com/octoleo/lsyncd.git
synced 2024-12-12 14:17:47 +00:00
Update docs
This commit is contained in:
Daniel Poelzleithner
parent
f244405bce
commit
e94edac51e
@ -32,8 +32,20 @@ This is an optimization which can be dangerous; so, please use it only if you ar
|
||||
|
||||
The default behaviors you can select from are following:
|
||||
|
||||
Shared Settings
|
||||
---------------
|
||||
|
||||
Following settings are shared between all defaults:
|
||||
|
||||
| Name | Description |
|
||||
|-----------------|------------------|
|
||||
| source | Source directory |
|
||||
|
||||
|
||||
|
||||
default.rsync
|
||||
-------------
|
||||
|
||||
The default rsync configuration will aggregate events up to ```delay``` seconds or 1000 separate uncollapsible events, which ever happens first. Then it will spawn one Rsync with a filter of all files that changed. The filter list is transmitted to Rsync trough a pipe. A call from Lsyncd to Rsync will thus look like this:
|
||||
|
||||
{% highlight shell %}
|
||||
@ -58,6 +70,12 @@ sync {
|
||||
}
|
||||
{% endhighlight %}
|
||||
|
||||
Additional settings:
|
||||
| Name | Description |
|
||||
|-----------------|-------------|
|
||||
| batchSizeLimit | Files larger then this limit should not be batched into on transfer. Only makes sense with processes > 1 which prevents rsyncssh |
|
||||
|
||||
|
||||
Below is a table of options for the ```rsync``` parameter. Please have a look at the Rsync documentation for an in depth explanation.
|
||||
|
||||
<table>
|
||||
@ -393,6 +411,7 @@ Below is a table of options for the ```rsync``` parameter. Please have a look at
|
||||
|
||||
default.rsyncssh
|
||||
----------------
|
||||
|
||||
This configuration differs from the standard rsync configuration in that it uses ssh commands to move files or directories locally at the target host instead of deleting and transferring again. This configuration does spawn Rsync processes like default.rsync but additionally will spawn ```/usr/bin/ssh HOST mv ORIGIN DESTINATION``` commands.
|
||||
|
||||
Different to default.rsync it does not take an uniform ```target``` parameter, but needs ```host``` and ```targetdir``` separated.
|
||||
@ -463,8 +482,8 @@ sync {
|
||||
|
||||
Please note the comma between the ```rsync``` parameter set and the ```ssh``` parameter set.
|
||||
|
||||
**Caution**
|
||||
If you are upgrading from 2.0.x, please notice that `settings` became a function from a variable, so you **MUST** delete the equal sign '=' between `settings` and the `{`.
|
||||
__Caution__
|
||||
If you are upgrading from 2.0.x, please notice that `settings` became a function from a variable, so you __MUST__ delete the equal sign '=' between `settings` and the `{`.
|
||||
|
||||
Lsyncd will call ```xargs``` on the remote host to handle multiple tasks in a single connection. Xargs options can be specified by the xargs parameter.
|
||||
|
||||
@ -490,7 +509,6 @@ Lsyncd will call ```xargs``` on the remote host to handle multiple tasks in a si
|
||||
|
||||
</table>
|
||||
|
||||
|
||||
Example:
|
||||
|
||||
{% highlight lua %}
|
||||
@ -504,6 +522,7 @@ sync {
|
||||
|
||||
default.direct
|
||||
-------------
|
||||
|
||||
Default.direct can be used to keep two local directories in sync with better performance than using default.rsync. Default.direct uses (just like default.rsync) rsync on startup to initially synchronize the target directory with the source directory. However, during normal operation default.direct uses /bin/cp, /bin/rm and /bin/mv to keep the synchronization. All parameters are just like default.rsync.
|
||||
|
||||
Example:
|
||||
@ -518,6 +537,7 @@ sync {
|
||||
|
||||
Exclusions
|
||||
----------
|
||||
|
||||
Two additional parameters can be specified to sync{}:
|
||||
|
||||
<table>
|
||||
@ -538,12 +558,12 @@ Two additional parameters can be specified to sync{}:
|
||||
|
||||
Exclusion rules are modeled after rsync's exclusion patterns but are a bit simpler. Lsyncd supports these features:
|
||||
|
||||
* Generally if any segment of the pathname (see below Layer 3) of an event matches the text, it is excluded. E.g. the file "/bin/foo/bar" matches the rule "foo".
|
||||
* If the rule starts with a slash, it will only be matched at the beginning of the pathname
|
||||
* If the rule ends with a slash, it will only be matched at the end of a pathname
|
||||
* ? matches any character that is not a slash.
|
||||
* ```*``` matches zero or more characters that are not a slash
|
||||
* ```**``` matches zero or more characters, this can be slashes.
|
||||
* Generally if any segment of the pathname (see below Layer 3) of an event matches the text, it is excluded. E.g. the file "/bin/foo/bar" matches the rule "foo".
|
||||
* If the rule starts with a slash, it will only be matched at the beginning of the pathname
|
||||
* If the rule ends with a slash, it will only be matched at the end of a pathname
|
||||
* ? matches any character that is not a slash.
|
||||
* ```*``` matches zero or more characters that are not a slash
|
||||
* ```**``` matches zero or more characters, this can be slashes.
|
||||
|
||||
Example:
|
||||
|
||||
@ -552,12 +572,13 @@ sync {
|
||||
default.rsync,
|
||||
source = "/home/user/src/",
|
||||
targetdir = "/home/user/dst/",
|
||||
exclude = { '*.bak' , '*.tmp' }
|
||||
exclude = { '_.bak' , '_.tmp' }
|
||||
}
|
||||
{% endhighlight %}
|
||||
|
||||
Deletions
|
||||
---------
|
||||
|
||||
By default Lsyncd will delete files on the target that are not present at the source since this is a fundamental part of the idea of keeping the target in sync with the source. However, many users requested exceptions for this, for various reasons, so all default implementations take ```delete``` as an additional parameter.
|
||||
|
||||
Valid values for ```delete``` are:
|
||||
@ -589,3 +610,87 @@ Valid values for ```delete``` are:
|
||||
</td></tr>
|
||||
|
||||
</table>
|
||||
|
||||
Tunnels
|
||||
-------
|
||||
|
||||
*New in: 2.3.0*
|
||||
|
||||
Lsyncd is able to start and manage external programs to provide a tunnel for data transfer.
|
||||
Additionally it can spawn multiple connections and load-balance connections among them. A tunnel is created through the `tunnel` function.
|
||||
|
||||
```
|
||||
{% highlight lua %}
|
||||
sync {
|
||||
default.rsync,
|
||||
tunnel = tunnel {
|
||||
command = {"ssh", "-N", "-L", "localhost:5432:localhost:873", "tunnel@testmachine"},
|
||||
}
|
||||
target = "rsync://localhost:5432/projects",
|
||||
source = "/home/user/src/",
|
||||
}
|
||||
{% endhighlight %}
|
||||
```
|
||||
|
||||
You can then set the shell for the tunnel user to `/bin/false` and configure the rsyncd server side appropriately.
|
||||
|
||||
Valid arguments for `tunnel` are:
|
||||
|
||||
| Argument | Description | Default | Valid values |
|
||||
|------------|-------------------------------------------------------|-----------|-------------------------------|
|
||||
| mode | Mode in which the tunnel is run | command | command, pool |
|
||||
| command | _Required_ Command to run | nil | Table of arguments |
|
||||
| parallel | How many connections to run. Only for pool mode | 1 | 1+ |
|
||||
| retryDelay | Seconds to wait until tunnel is restarted | 10 | Number |
|
||||
| readyDelay | Seconds after program start to consider tunnel up | 5 | Number |
|
||||
| localhost | Name of the local host variable | localhost | String |
|
||||
|
||||
## Pool Mode
|
||||
|
||||
In pool mode, lsyncd allocates a new local port which is then passed as variable to the host command.
|
||||
All variables can be substituded by `^variable` syntax.
|
||||
See [../../../../examples/lrsyncssh-tunnel.lua](lrsyncssh-tunnel.lua) for a extended configuration.
|
||||
|
||||
### List of variables
|
||||
|
||||
| Name | Description |
|
||||
|------------|-----------------------------------------------|
|
||||
| localport | Port allocated for the selected connection |
|
||||
| localhost | Local hostname used. Default localhost |
|
||||
|
||||
### Example
|
||||
|
||||
Run 2 tunnel ssh processes and 4 rsync processes at the same time. Use extra transfers for all files larger 30 MB.
|
||||
|
||||
```lua
|
||||
sync {
|
||||
default.rsync,
|
||||
tunnel = tunnel {
|
||||
command = {"ssh", "-N", "-L", "localhost:^localport:localhost:873", "user@testmachine"},
|
||||
mode = "pool",
|
||||
parallel = 2,
|
||||
},
|
||||
source = "/data/projects",
|
||||
target = "rsync://localhost:^localport/projects",
|
||||
delay = 5,
|
||||
batchSizeLimit = 1024 * 1024 * 30,
|
||||
maxProcesses = 4
|
||||
}
|
||||
```
|
||||
|
||||
This example will open 2 ssh connections for port forwarding and load balance 4 parallel rsync
|
||||
processes in a round roubin fashion.
|
||||
|
||||
### Workflow in Poolmodel
|
||||
|
||||
When a sync with a tunnel parameter is started, all events are queued until the tunnel reaches the
|
||||
`UP` state, which is when one successful tunnel process exists for at least `readyDelay` seconds.
|
||||
Dead tunnel processes are automatically restarted. When the tunnel process count drops to 0, tunnel falls back to the
|
||||
`CONNECTING` state. There is a `retryDelay` seconds delay between each attempt to restart the tunnel.
|
||||
|
||||
Once the tunnel is UP, a full transfer is initiated. Subsequent transfers are then load balanced over multiple connections.
|
||||
|
||||
### Notes on Poolmode
|
||||
|
||||
* Pool mode only works with `rsync` backend, since there is no way to prevent multiple transfers to the same file in a relieable way, the rsync backend only supports `maxProcesses = 1` which renders pool mode useless. Since the remote side rsync daemon can prevent file trashing, the rsync backend is safe.
|
||||
|
||||
|
||||
@ -4,73 +4,87 @@ title: Invoking
|
||||
---
|
||||
As most Unix tools, Lsyncd will print a synopsis of its command line options when called with --help.
|
||||
|
||||
{% highlight shell %}
|
||||
```console
|
||||
lsyncd --help
|
||||
lsyncd -help
|
||||
{% endhighlight %}
|
||||
```
|
||||
|
||||
The two hyphens are redundant for Lsyncd. It has no short one letter options and one hyphen will always result into the same as specifying two.
|
||||
|
||||
Also like most Unix tools, ```--version``` or ```-version``` will let Lsyncd print its version number.
|
||||
|
||||
{% highlight shell %}
|
||||
```console
|
||||
lsyncd -version
|
||||
{% endhighlight %}
|
||||
```
|
||||
|
||||
Lsyncd 2.1 is designed to be predominantly configured through a config file (see below). The config file can thus be the only command line option.
|
||||
|
||||
{% highlight shell %}
|
||||
```console
|
||||
lsyncd CONFIGFILE
|
||||
{% endhighlight %}
|
||||
```
|
||||
|
||||
Although for standard use or quick testing it can be cursorily configured by command line options. The following will keep a local source and destination directory in sync using rsync:
|
||||
|
||||
{% highlight shell %}
|
||||
```console
|
||||
lsyncd -rsync /home/USER/src /home/USER/dst
|
||||
{% endhighlight %}
|
||||
```
|
||||
|
||||
The target can here be anything that Rsync recognizes.
|
||||
|
||||
{% highlight shell %}
|
||||
```console
|
||||
lsyncd -rsync /home/USER/src remotehost:dst
|
||||
{% endhighlight %}
|
||||
```
|
||||
|
||||
Two (or more) targets are configured by calling -rsync twice (or several times).
|
||||
|
||||
{% highlight shell %}
|
||||
```console
|
||||
lsyncd -rsync /home/USER/src remotehost1:dst -rsync /home/USER/src remotehost2:dst
|
||||
{% endhighlight %}
|
||||
```
|
||||
|
||||
A disadvantage with Rsync synchronization is that normally directory and file moves result in a deletion of the move origin and a retransfer of the move destination of the wire. However, Lsyncd 2 can use ssh commands to move the directories and files locally on the target. To use this use ```-rsyncssh``` followed by the local source directory, the remote host and the target directory there. The REMOTEHOST can include a user like ```me@remotehost.com```.
|
||||
|
||||
{% highlight shell %}
|
||||
```console
|
||||
lsyncd -rsyncssh /home/USER/src REMOTEHOST TARGETDIR
|
||||
{% endhighlight %}
|
||||
```
|
||||
|
||||
When testing Lsyncd configurations ```-nodaemon``` is a pretty handy flag. With this option, Lsyncd will not detach and will not become a daemon. All log messages are additionally to the configured logging facilities printed on the console (_stdout_ and _stderr_).
|
||||
|
||||
{% highlight shell %}
|
||||
lsyncd -nodaemon CONFIGFILE
|
||||
{% endhighlight %}
|
||||
|
||||
There is a difference in behaviour when running with -nodaemon. Lsyncd will not change its working directory to ```/```, as it does when becoming a daemon. Thus relative targets like ```./target``` will work with ```-nodaemon``` but must be specified to absolute paths to work in daemon mode. The source directories will also be turned into absolute paths by Lsyncd. The reason targets are not resolved to absolute paths while sources are is because Lsyncd itself does not care about the format of the target specifier which can also be remote hosts, rsyncd modules, etc. It is opaquely handed to rsync. It cares about the observed directories though.
|
||||
```console
|
||||
lsyncd -nodaemon CONFIGFILE
|
||||
```
|
||||
|
||||
There is a difference in behaviour when running with -nodaemon. Lsyncd will not change its working directory to `/`, as it does when becoming a daemon. Thus relative targets like `./target` will work with `-nodaemon` but must be specified to absolute paths to work in daemon mode. The source directories will also be turned into absolute paths by Lsyncd. The reason targets are not resolved to absolute paths while sources are is because Lsyncd itself does not care about the format of the target specifier which can also be remote hosts, rsyncd modules, etc. It is opaquely handed to rsync. It cares about the observed directories though.
|
||||
|
||||
|
||||
*New in 2.3.0*
|
||||
|
||||
To only trigger the initial full sync and not monitor the file system, you can use the `-onepass` option.
|
||||
|
||||
```console
|
||||
lsyncd -onepass CONFIGFILE
|
||||
```
|
||||
|
||||
|
||||
## Logging
|
||||
|
||||
|
||||
All log messages are sorted in categories. By default Lsyncd is scarce with log messages. You can turn Lsyncd into a motormouth by specifying ```-log all```.
|
||||
|
||||
{% highlight shell %}
|
||||
```console
|
||||
lsyncd -log all CONFIGFILE
|
||||
{% endhighlight %}
|
||||
```
|
||||
|
||||
This might easily become too much. A particularly useful category is "Exec" which will log the command lines of all processes Lsyncd spawns.
|
||||
|
||||
{% highlight shell %}
|
||||
```console
|
||||
lsyncd -log Exec CONFIGFILE
|
||||
{% endhighlight %}
|
||||
```
|
||||
|
||||
When the initial startup sync fails by default Lsyncd will terminate with an error message. It has been designed this way, so configuration failures are visibly reported to a possibly beginning user. However, in production a remote target might be done, but you want Lsyncd to start nevertheless and keep trying to sync to the remote target until it is up.
|
||||
|
||||
{% highlight shell %}
|
||||
```console
|
||||
lsyncd -insist -rsync /home/USER/src remotehost:dst
|
||||
{% endhighlight %}
|
||||
```
|
||||
|
||||
In production mode it is recommended to have insist on. It can also be specified in the settings{} command in a config file.
|
||||
|
||||
Loading…
Reference in New Issue
Block a user