octoleo/syncthing
1
0
Fork 0
mirror of https://github.com/octoleo/syncthing.git synced 2024-11-09 14:50:56 +00:00
Code Issues Packages Projects Releases Wiki Activity

Move protocol specs to a separate repo

Browse Source
This commit is contained in:
Jakob Borg 2014-12-22 09:53:02 +01:00
parent 4b76ec40c0
commit 7344a6205f
4 changed files with 3 additions and 919 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

9
README.md
View File

@ -11,7 +11,7 @@ This is the `syncthing` project. The following are the project goals:
collaborating devices. The protocol should be well defined, unambiguous, collaborating devices. The protocol should be well defined, unambiguous,
easily understood, free to use, efficient, secure and language neutral. easily understood, free to use, efficient, secure and language neutral.
This is the [Block Exchange This is the [Block Exchange
Protocol](https://github.com/syncthing/syncthing/blob/master/protocol/PROTOCOL.md). Protocol](https://github.com/syncthing/protocol/blob/master/BEPv1.md).
2. Provide the reference implementation to demonstrate the usability of 2. Provide the reference implementation to demonstrate the usability of
said protocol. This is the `syncthing` utility. It is the hope that said protocol. This is the `syncthing` utility. It is the hope that
@ -55,13 +55,6 @@ The [syncthing
documentation](http://discourse.syncthing.net/category/documentation) is documentation](http://discourse.syncthing.net/category/documentation) is
on the discourse site. on the discourse site.
License
=======
All documentation and protocol specifications are licensed
under the [Creative Commons Attribution 4.0 International
License](http://creativecommons.org/licenses/by/4.0/).
All code is licensed under the All code is licensed under the
[GPL](https://github.com/syncthing/syncthing/blob/master/LICENSE), v3 or [GPL](https://github.com/syncthing/syncthing/blob/master/LICENSE), v3 or
later. later.

188
protocol/DISCOVERY.md
View File

@ -1,188 +0,0 @@
Device Discovery Protocol v2
============================
Mode of Operation
-----------------
There are two distinct modes: **local discovery**, performed on a LAN
segment (broadcast domain) and **global discovery**, performed over the
Internet with the support of a well-known server. Both modes are over UDP.
Local discovery
---------------
Each participating device sends periodically an Announcement packet and keeps
a table of the announcements it has seen. There is no way to solicit a reply;
the only message type is Announcement.
On multihomed hosts the announcement packets should be sent on each
interface on which syncthing will accept connections.
For IPv4, the Announcement packet is broadcasted either to the link-specific
broadcast address, or to the generic link-local broadcast address
`255.255.255.255`, with source and destination port 21025.
For IPv6, the Announcement packet is multicasted to the transient link-local
multicast address `[ff32::5222]`, with source and destination port 21026.
It is recommended that local discovery Announcement packets be sent on
a 30 to 60 second interval, possibly with immediate transmissions when a
previously unknown device is discovered.
Global discovery
----------------
Global discovery is performed in two steps: announcement and discovery.
In the announcement step, a device periodically unicasts an Announcement
packet to the global server `announce.syncthing.net`, port 22026.
In the discovery step, a device sends a Query packet for a given device ID to
the server. If the server knows the ID, it replies with an Announcement packet.
If the server doesn't know the ID, it will not reply. The device must interpret
a timeout as lookup failure.
There is no message to unregister from the server; instead the server forgets
about an Announcement after 60 minutes.
It is recommended to send Announcement packets to the global server on a 30
minute interval.
Device ID
---------
The device ID is the SHA-256 (32 bytes) of the device X.509 certificate.
See [How device IDs work] in the Syncthing documentation.
Announcement packet
-------------------
The Announcement packet has the following structure:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Magic (0x9D79BC39) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
/ /
\ Device Structure \
/ /
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Number of Extra Devices |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
/ /
\ Zero or more Device Structures \
/ /
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Device Structure:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Length of Device ID |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
/ /
\ Device ID \
/ /
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Number of Addresses |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
/ /
\ Zero or more Address Structures \
/ /
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Address Structure:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Length of IP |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
/ /
\ IP (variable length) \
/ /
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Port |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
The corresponding XDR representation is as follows (see [RFC4506] for the XDR
format):
struct Announcement {
unsigned int Magic;
Device This;
Device Extra<>;
}
struct Device {
opaque DeviceID<32>;
Address Addresses<>;
}
struct Address {
opaque IP<>;
unsigned int Port;
}
The first Device structure contains information about the sending device.
The following zero or more Extra devices contain information about other
devices known to the sending device.
In the `Device` structure, field `DeviceID` is the SHA-256 (32 bytes) of the
device X.509 certificate, as explained in section _Device ID_.
In the `Address` structure, the IP field can be of three different kinds:
- A zero length indicates that the IP address should be taken from the
source address of the announcement packet, be it IPv4 or IPv6. The
source address must be a valid unicast address. This is only valid
in the first device structure, not in the list of extras. In case of
global discovery, the discovery server will reply to a Query with an
announcement packet containing the expanded address of the queried
device ID as seen from the server, allowing to traverse the majority of
NAT devices.
- A four byte length indicates that the address is an IPv4 unicast
address.
- A sixteen byte length indicates that the address is an IPv6 unicast
address.
Although the `port` field is 32-bit, the value should fit in a 16-bit unsigned
integer, since that is the size of a UDP port.
Query packet
------------
The Query packet has the following structure:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Magic Number (0x2CA856F5) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Length of Device ID |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
/ /
\ Device ID \
/ /
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
This is the XDR encoding of:
struct Query {
unsigned int MagicNumber;
opaque DeviceID<32>;
}
Design rationale
----------------
At the beginning, also IPv4 was using multicast. It has been changed to
broadcast after some bugs, especially on Android.
[How device IDs work]: https://discourse.syncthing.net/t/how-device-ids-work/365
[RFC4506]: http://tools.ietf.org/html/rfc4506

723
protocol/PROTOCOL.md
View File

@ -1,723 +0,0 @@
Block Exchange Protocol v1
==========================
Introduction and Definitions
----------------------------
BEP is used between two or more _devices_ thus forming a _cluster_. Each
device has one or more _folders_ of files described by the _local
model_, containing metadata and block hashes. The local model is sent to
the other devices in the cluster. The union of all files in the local
models, with files selected for highest change version, forms the
_global model_. Each device strives to get its folders in sync with
the global model by requesting missing or outdated blocks from the other
devices in the cluster.
File data is described and transferred in units of _blocks_, each being
128 KiB (131072 bytes) in size.
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL
NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and
"OPTIONAL" in this document are to be interpreted as described in
RFC 2119.
Transport and Authentication
----------------------------
BEP is deployed as the highest level in a protocol stack, with the lower
level protocols providing encryption and authentication.
+-----------------------------|
| Block Exchange Protocol |
|-----------------------------|
| Encryption & Auth (TLS 1.2) |
|-----------------------------|
| TCP |
|-----------------------------|
v ... v
The encryption and authentication layer SHALL use TLS 1.2 or a higher
revision. A strong cipher suite SHALL be used, with "strong cipher
suite" being defined as being without known weaknesses and providing
Perfect Forward Secrecy (PFS). Examples of strong cipher suites are
given at the end of this document. This is not to be taken as an
exhaustive list of allowed cipher suites but represents best practices
at the time of writing.
The exact nature of the authentication is up to the application, however
it SHALL be based on the TLS certificate presented at the start of the
connection. Possibilities include certificates signed by a common
trusted CA, preshared certificates, preshared certificate fingerprints
or certificate pinning combined with some out of band first
verification. The reference implementation uses preshared certificate
fingerprints (SHA-256) referred to as "Device IDs".
There is no required order or synchronization among BEP messages except
as noted per message type - any message type may be sent at any time and
the sender need not await a response to one message before sending
another. Responses MUST however be sent in the same order as the
requests are received.
The underlying transport protocol MUST be TCP.
Messages
--------
Every message starts with one 32 bit word indicating the message
version, type and ID, followed by the length of the message. The header
is in network byte order, i.e. big endian.
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Ver | Message ID | Type | Reserved |C|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Length |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
For BEP v1 the Version field is set to zero. Future versions with
incompatible message formats will increment the Version field. A message
with an unknown version is a protocol error and MUST result in the
connection being terminated. A client supporting multiple versions MAY
retry with a different protocol version upon disconnection.
The Message ID is set to a unique value for each transmitted request
message. In response messages it is set to the Message ID of the
corresponding request message. The uniqueness requirement implies that
no more than 4096 messages may be outstanding at any given moment. The
ordering requirement implies that a response to a given message ID also
means that all preceding messages have been received, specifically those
which do not otherwise demand a response. Hence their message ID:s may
be reused.
The Type field indicates the type of data following the message header
and is one of the integers defined below. A message of an unknown type
is a protocol error and MUST result in the connection being terminated.
The Compression bit "C" indicates the compression used for the message.
For C=1:
* The Length field contains the length, in bytes, of the
compressed message data plus a four byte uncompressed length field.
* The compressed message data is preceeded by a 32 bit field denoting
the length of the uncompressed message.
* The message data is compressed using the LZ4 format and algorithm
described in https://code.google.com/p/lz4/.
For C=0:
* The Length field contains the length, in bytes, of the
uncompressed message data.
* The message is not compressed.
All data within the message (post decompression, if compression is
in use) MUST be in XDR (RFC 1014) encoding. All fields shorter than 32
bits and all variable length data MUST be padded to a multiple of 32
bits. The actual data types in use by BEP, in XDR naming convention, are
the following:
- (unsigned) int -- (unsigned) 32 bit integer
- (unsigned) hyper -- (unsigned) 64 bit integer
- opaque<> -- variable length opaque data
- string<> -- variable length string
The transmitted length of string and opaque data is the length of actual
data, excluding any added padding. The encoding of opaque<> and string<>
are identical, the distinction being solely one of interpretation.
Opaque data should not be interpreted but can be compared bytewise to
other opaque data. All strings MUST use the Unicode UTF-8 encoding,
normalization form C.
### Cluster Config (Type = 0)
This informational message provides information about the cluster
configuration as it pertains to the current connection. A Cluster Config
message MUST be the first message sent on a BEP connection. Additional
Cluster Config messages MUST NOT be sent after the initial exchange.
#### Graphical Representation
ClusterConfigMessage Structure:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Length of ClientName |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
/ /
\ ClientName (variable length) \
/ /
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Length of ClientVersion |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
/ /
\ ClientVersion (variable length) \
/ /
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Number of Folders |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
/ /
\ Zero or more Folder Structures \
/ /
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Number of Options |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
/ /
\ Zero or more Option Structures \
/ /
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Folder Structure:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Length of ID |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
/ /
\ ID (variable length) \
/ /
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Number of Devices |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
/ /
\ Zero or more Device Structures \
/ /
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Device Structure:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Length of ID |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
/ /
\ ID (variable length) \
/ /
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Flags |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
+ Max Local Version (64 bits) +
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Option Structure:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Length of Key |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
/ /
\ Key (variable length) \
/ /
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Length of Value |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
/ /
\ Value (variable length) \
/ /
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
#### Fields
The ClientName and ClientVersion fields identify the implementation. The
values SHOULD be simple strings identifying the implementation name, as
a user would expect to see it, and the version string in the same
manner. An example ClientName is "syncthing" and an example
ClientVersion is "v0.7.2". The ClientVersion field SHOULD follow the
patterns laid out in the [Semantic Versioning](http://semver.org/)
standard.
The Folders field lists all folders that will be synchronized
over the current connection. Each folder has a list of participating
Devices. Each device has an associated Flags field to indicate the sharing
mode of that device for the folder in question. See the discussion on
Sharing Modes.
The Device Flags field contains the following single bit flags:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Reserved |Pri| Reserved |I|R|T|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
- Bit 31 ("T", Trusted) is set for devices that participate in trusted
mode.
- Bit 30 ("R", Read Only) is set for devices that participate in read
only mode.
- Bit 29 ("I", Introducer) is set for devices that are trusted as cluster
introducers.
- Bits 16 through 28 are reserved and MUST be set to zero.
- Bits 14-15 ("Pri) indicate the device's upload priority for this
folder. Possible values are:
- 00: The default. Normal priority.
- 01: High priority. Other devices SHOULD favour requesting files from
this device over devices with normal or low priority.
- 10: Low priority. Other devices SHOULD avoid requesting files from
this device when they are available from other devices.
- 11: Sharing disabled. Other devices SHOULD NOT request files from
this device.
- Bits 0 through 14 are reserved and MUST be set to zero.
Exactly one of the T and R bits MUST be set.
The per device Max Local Version field contains the highest local file
version number of the files already known to be in the index sent by
this device. If nothing is known about the index of a given device, this
field MUST be set to zero. When receiving a Cluster Config message with
a non-zero Max Local Version for the local device ID, a device MAY elect to
send an Index Update message containing only files with higher local
version numbers in place of the initial Index message.
The Options field contain option values to be used in an implementation
specific manner. The options list is conceptually a map of Key => Value
items, although it is transmitted in the form of a list of (Key, Value)
pairs, both of string type. Key ID:s are implementation specific. An
implementation MUST ignore unknown keys. An implementation MAY impose
limits on the length keys and values. The options list may be used to
inform devices of relevant local configuration options such as rate
limiting or make recommendations about request parallelism, device
priorities, etc. An empty options list is valid for devices not having any
such information to share. Devices MAY NOT make any assumptions about
peers acting in a specific manner as a result of sent options.
#### XDR
struct ClusterConfigMessage {
string ClientName<>;
string ClientVersion<>;
Folder Folders<>;
Option Options<>;
}
struct Folder {
string ID<>;
Device Devices<>;
}
struct Device {
string ID<>;
unsigned int Flags;
unsigned hyper MaxLocalVersion;
}
struct Option {
string Key<>;
string Value<>;
}
### Index (Type = 1) and Index Update (Type = 6)
The Index and Index Update messages define the contents of the senders
folder. An Index message represents the full contents of the
folder and thus supersedes any previous index. An Index Update
amends an existing index with new information, not affecting any entries
not included in the message. An Index Update MAY NOT be sent unless
preceded by an Index, unless a non-zero Max Local Version has been
announced for the given folder by the peer device.
An Index or Index Update message MUST be sent for each folder
included in the Cluster Config message, and MUST be sent before any
other message referring to that folder. A device with no data to
advertise MUST send an empty Index message (a file list of zero length).
If the folder contents change from non-empty to empty, an empty
Index message MUST be sent. There is no response to the Index message.
#### Graphical Representation
IndexMessage Structure:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Length of Folder |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
/ /
\ Folder (variable length) \
/ /
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Number of Files |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
/ /
\ Zero or more FileInfo Structures \
/ /
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
FileInfo Structure:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Length of Name |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
/ /
\ Name (variable length) \
/ /
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Flags |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
+ Modified (64 bits) +
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
+ Version (64 bits) +
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
+ Local Version (64 bits) +
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Number of Blocks |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
/ /
\ Zero or more BlockInfo Structures \
/ /
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
BlockInfo Structure:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Size |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Length of Hash |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
/ /
\ Hash (variable length) \
/ /
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
#### Fields
The Folder field identifies the folder that the index message
pertains to. For single folder implementations the device MAY send an
empty folder ID or use the string "default".
The Name is the file name path relative to the folder root. Like all
strings in BEP, the Name is always in UTF-8 NFC regardless of operating
system or file system specific conventions. The Name field uses the
slash character ("/") as path separator, regardless of the
implementation's operating system conventions. The combination of
Folder and Name uniquely identifies each file in a cluster.
The Version field is the value of a cluster wide Lamport clock
indicating when the change was detected. The clock ticks on every
detected and received change. The combination of Folder, Name and
Version uniquely identifies the contents of a file at a given point in
time.
The Local Version field is the value of a device local monotonic clock at
the time of last local database update to a file. The clock ticks on
every local database update.
The Flags field is made up of the following single bit flags:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Reserved |U|S|P|I|D| Unix Perm. & Mode |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
- The lower 12 bits hold the common Unix permission and mode bits. An
implementation MAY ignore or interpret these as is suitable on the host
operating system.
- Bit 19 ("D") is set when the file has been deleted. The block list
SHALL be of length zero and the modification time indicates the time
of deletion or, if the time of deletion is not reliably determinable,
the last known modification time.
- Bit 18 ("I") is set when the file is invalid and unavailable for
synchronization. A peer MAY set this bit to indicate that it can
temporarily not serve data for the file.
- Bit 17 ("P") is set when there is no permission information for the
file. This is the case when it originates on a non-permission-
supporting file system. Changes to only permission bits SHOULD be
disregarded on files with this bit set. The permissions bits MUST be
set to the octal value 0666.
- Bit 16 ("S") is set when the file is a symbolic link. The block list
SHALL be of one or more blocks since the target of the symlink is
stored within the blocks of the file.
- Bit 15 ("U") is set when the symbolic links target does not exist.
On systems where symbolic links have types, this bit being means
that the default file symlink SHALL be used. If this bit is unset
bit 19 will decide the type of symlink to be created.
- Bit 0 through 14 are reserved for future use and SHALL be set to
zero.
The hash algorithm is implied by the Hash length. Currently, the hash
MUST be 32 bytes long and computed by SHA256.
The Modified time is expressed as the number of seconds since the Unix
Epoch (1970-01-01 00:00:00 UTC).
In the rare occasion that a file is simultaneously and independently
modified by two devices in the same cluster and thus end up on the same
Version number after modification, the Modified field is used as a tie
breaker (higher being better), followed by the hash values of the file
blocks (lower being better).
The Blocks list contains the size and hash for each block in the file.
Each block represents a 128 KiB slice of the file, except for the last
block which may represent a smaller amount of data.
#### XDR
struct IndexMessage {
string Folder<>;
FileInfo Files<>;
}
struct FileInfo {
string Name<>;
unsigned int Flags;
hyper Modified;
unsigned hyper Version;
unsigned hyper LocalVer;
BlockInfo Blocks<>;
}
struct BlockInfo {
unsigned int Size;
opaque Hash<>;
}
### Request (Type = 2)
The Request message expresses the desire to receive a data block
corresponding to a part of a certain file in the peer's folder.
#### Graphical Representation
RequestMessage Structure:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Length of Folder |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
/ /
\ Folder (variable length) \
/ /
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Length of Name |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
/ /
\ Name (variable length) \
/ /
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
+ Offset (64 bits) +
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Size |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
#### Fields
The Folder and Name fields are as documented for the Index message.
The Offset and Size fields specify the region of the file to be
transferred. This SHOULD equate to exactly one block as seen in an Index
message.
#### XDR
struct RequestMessage {
string Folder<>;
string Name<>;
unsigned hyper Offset;
unsigned int Size;
}
### Response (Type = 3)
The Response message is sent in response to a Request message.
#### Graphical Representation
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Length of Data |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
/ /
\ Data (variable length) \
/ /
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
#### Fields
The Data field contains either a full 128 KiB block, a shorter block in
the case of the last block in a file, or is empty (zero length) if the
requested block is not available.
#### XDR
struct ResponseMessage {
opaque Data<>
}
### Ping (Type = 4)
The Ping message is used to determine that a connection is alive, and to
keep connections alive through state tracking network elements such as
firewalls and NAT gateways. The Ping message has no contents.
### Pong (Type = 5)
The Pong message is sent in response to a Ping. The Pong message has no
contents, but copies the Message ID from the Ping.
### Close (Type = 7)
The Close message MAY be sent to indicate that the connection will be
torn down due to an error condition. A Close message MUST NOT be
followed by further messages.
#### Graphical Representation
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Length of Reason |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
/ /
\ Reason (variable length) \
/ /
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
#### Fields
The Reason field contains a human description of the error condition,
suitable for consumption by a human.
struct CloseMessage {
string Reason<1024>;
}
Sharing Modes
-------------
### Trusted
Trusted mode is the default sharing mode. Updates are exchanged in both
directions.
+------------+ Updates /---------\
| | -----------> / \
| Device | | Cluster |
| | <----------- \ /
+------------+ Updates \---------/
### Read Only
In read only mode, a device does not synchronize the local folder to
the cluster, but publishes changes to its local folder contents as
usual. The local folder can be seen as a "master copy" that is never
affected by the actions of other cluster devices.
+------------+ Updates /---------\
| | -----------> / \
| Device | | Cluster |
| | \ /
+------------+ \---------/
Message Limits
--------------
An implementation MAY impose reasonable limits on the length of message
fields to aid robustness in the face of corruption or broken
implementations. These limits, if imposed, SHOULD NOT be more
restrictive than the following:
### Index and Index Update Messages
- Folder: 64 bytes
- Number of Files: 10.000.000
- Name: 1024 bytes
- Number of Blocks: 1.000.000
- Hash: 64 bytes
### Request Messages
- Folder: 64 bytes
- Name: 1024 bytes
### Response Messages
- Data: 256 KiB
### Options Message
- Number of Options: 64
- Key: 64 bytes
- Value: 1024 bytes
Example Exchange
----------------
A B
1. Index-> <-Index
2. Request->
3. Request->
4. Request->
5. Request->
6. <-Response
7. <-Response
8. <-Response
9. <-Response
10. Index Update->
...
11. Ping->
12. <-Pong
The connection is established and at 1. both peers send Index records.
The Index records are received and both peers recompute their knowledge
of the data in the cluster. In this example, peer A has four missing or
outdated blocks. At 2 through 5 peer A sends requests for these blocks.
The requests are received by peer B, who retrieves the data from the
folder and transmits Response records (6 through 9). Device A updates
their folder contents and transmits an Index Update message (10).
Both peers enter idle state after 10. At some later time 11, peer A
determines that it has not seen data from B for some time and sends a
Ping request. A response is sent at 12.
Examples of Strong Cipher Suites
--------------------------------
* 0x009F DHE-RSA-AES256-GCM-SHA384 (TLSv1.2 DH RSA AESGCM(256) AEAD)
* 0x006B DHE-RSA-AES256-SHA256 (TLSv1.2 DH RSA AES(256) SHA256)
* 0xC030 ECDHE-RSA-AES256-GCM-SHA384 (TLSv1.2 ECDH RSA AESGCM(256) AEAD)
* 0xC028 ECDHE-RSA-AES256-SHA384 (TLSv1.2 ECDH RSA AES(256) SHA384)
* 0x009E DHE-RSA-AES128-GCM-SHA256 (TLSv1.2 DH RSA AESGCM(128) AEAD)
* 0x0067 DHE-RSA-AES128-SHA256 (TLSv1.2 DH RSA AES(128) SHA256)
* 0xC02F ECDHE-RSA-AES128-GCM-SHA256 (TLSv1.2 ECDH RSA AESGCM(128) AEAD)
* 0xC027 ECDHE-RSA-AES128-SHA256 (TLSv1.2 ECDH RSA AES(128) SHA256)

2
protocol/README.md Normal file
View File

@ -0,0 +1,2 @@
Syncthing uses the protocols defined in
https://github.com/syncthing/protocol/.