Commit Graph

531 Commits

Author SHA1 Message Date
Jakob Borg
7c7e8648ff lib/model: Trigger a puller iteration on connection (fixes #3451)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3453
2016-07-27 21:35:41 +00:00
Jakob Borg
66a506e72b lib/scanner: Correctly scan symlinks (fixes #3445)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3446
2016-07-26 11:55:25 +00:00
Jakob Borg
7aaa1dd8a3 lib/scanner: Recheck file size and modification time after hashing (ref #3440)
To catch the case where the file changed. Also make sure we never let a
size-vs-blocklist mismatch slip through.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3443
2016-07-26 08:51:39 +00:00
Jakob Borg
2a6f164923 lib/scanner: When scanning a file, stick to the size given by Lstat (fixes #3440)
Otherwise if the file grows during scanning the block list will be out
of sync with the stated size and things get confused. We could fixup the
size afterwards based on the block list, but then we might see other
inconsistencies as the mtime should have changed to reflect the new size
etc. Better stick to the original state and let the next scan pick up
the change.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3442
2016-07-25 19:16:49 +00:00
Jakob Borg
6ed22d0885 lib/model: Stricter temporary file permissions
We could have a file to sync with permissions rw------- but we'd create
the temp file with rw-rw-rw- minus umask, usually rw-r--r--. This
potentially exposes private data while the file is being synced.

Similarly, when ignorePerms was set and we were reusing a temp files we
would set the permissions to rw-r--r-- explicitly, potentially
overriding a strict umask that would otherwise have had the file be
rw-------.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3437
2016-07-25 10:18:05 +00:00
Jakob Borg
694da60659 lib/db: Reinstate database update locking
The previous commit loosened the locking around database updates.
Apparently that was not fine - what happens is that parallell updates
to the same file for different devices stomp on each others updates to
the global index, leaving it missing one of the two devices.
2016-07-23 20:32:15 +02:00
Jakob Borg
47fa4b0a2c cmd/syncthing, lib/db, lib/model, lib/protocol: Implement delta indexes (fixes #438)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3427
2016-07-23 12:46:31 +00:00
Jakob Borg
8ab6b60778 lib/model: Sort outgoing index updates by LocalVersion
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3411
2016-07-21 17:21:15 +00:00
Jakob Borg
7b7e35d339 lib/protocol: Hello message length is an int16
It used to be an int32, but that's unnecessary and the spec now says
int16. Also relaxes the size requirement to that which fits in a signed
int16 instead of limiting to 1024 bytes, to allow for future growth.

As reported in
https://forum.syncthing.net/t/difference-between-documented-and-implemented-protocol/7798

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3406
2016-07-17 21:41:20 +00:00
Jakob Borg
3176629410 cmd, lib: Fix ineffectual assignments (ineffasign) and comment spelling
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3405
2016-07-15 14:23:20 +00:00
Jakob Borg
e440d30028 lib/protocol: Allow unknown message types
This lets us add message types in the future, for authentication or
other purposes, without completely breaking old clients. I see this as
similar behavior to adding fields to messages - newer clients must
simple be aware that older ones may ignore the message and act
accordingly.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3390
2016-07-05 09:29:28 +00:00
Jakob Borg
44d30c83bf lib/config, cmd/syncthing: Handle committing configuration better (fixes #3077)
This slightly changes the interface used for committing configuration
changes. The two parts are now:

 - VerifyConfiguration, which runs synchronously and locked, and can
   abort the config change. These callbacks shouldn't *do* anything
   apart from looking at the config changes and saying yes or no. No
   change from previously.

 - CommitConfiguration, which runs asynchronously (one goroutine per
   call) *after* replacing the config and releasing any locks. Returning
   false from these methods sets the "requires restart" flag, which now
   lives in the config.Wrapper.

This should be deadlock free as the CommitConfiguration calls can take
as long as they like and can wait for locks to be released when they
need to tweak things. I think this should be safe compared to before as
the CommitConfiguration calls were always made from a random background
goroutine (typically one from the HTTP server), so it was always
concurrent with everything else anyway.

Hence the CommitResponse type is gone, instead you get an error back on
verification failure only, and need to explicitly check
w.RequiresRestart() afterwards if you care.

As an added bonus this fixes a bug where we would reset the "requires
restart" indicator if a config that did not require restart was saved,
even if we already were in the requires-restart state.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3386
2016-07-04 20:32:34 +00:00
Jakob Borg
dcb5026f33 script, lib/discover: Fixup copyright checks 2016-07-04 14:53:11 +02:00
Jakob Borg
9a0e5a7c18 lib/discover: Add instance ID to local discovery (fixes #3278)
A random "instance ID" is generated on each start of the local discovery
service. The instance ID is included in the announcement. When we see a
new instance ID we treat is a new device and respond with an
announcement of our own. Hence devices get to know each other quickly on
restart.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3385
2016-07-04 11:16:48 +00:00
Jakob Borg
aa50ef4069 lib/model: Invalidate files with trailing white space on Windows (fixes #3227)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3383
2016-07-04 10:44:30 +00:00
Jakob Borg
fa0101bd60 lib/protocol, lib/discover, lib/db: Use protocol buffer serialization (fixes #3080)
This changes the BEP protocol to use protocol buffer serialization
instead of XDR, and therefore also the database format. The local
discovery protocol is also updated to be protocol buffer format.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3276
LGTM: AudriusButkevicius
2016-07-04 10:40:29 +00:00
Cedric Staniewski
223a835f33 lib/discover: Respect the listen address scheme (fixes #3346)
This is a supplement patch to commit a58f69b which only fixed global
discovery. This patch adds the missing parts for the local discovery.

If the listen address scheme is set to tcp4:// or tcp6:// and no
explicit host is specified, an address should not be considered if the
source address does not match this scheme.

This prevents invalid URIs like tcp4://<IPv6 address>:<port> or tcp6://<IPv4
address>:<port> for local discovery.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3380
2016-07-03 20:43:26 +00:00
Jakob Borg
672824641b lib/connections: TLS handshake must complete in a timely fashion (fixes #3375)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3376
2016-07-02 20:33:31 +00:00
Jakob Borg
6d357211b2 lib/config: Remove "Invalid" attribute (fixes #2471)
This contains the following behavioral changes:

 - Duplicate folder IDs is now fatal during startup
 - Invalid folder flags in the ClusterConfig is fatal for the connection
   (this will go away soon with the proto changes, as we won't have any
   unknown flags any more then)
 - Empty path is a folder error reported at runtime

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3370
2016-07-02 19:38:39 +00:00
Nicolas Braud-Santoni
8e39e2889d lib: Fix typos in connections/service.go and model/model.go
Skip-check: authors

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3366
LGTM: aviau
2016-06-30 13:42:53 +00:00
Nicolas Braud-Santoni
a9ee4bb9f1 lib/upgrade: Remove TestGithubRelease (fixes #3362)
Skip-check: authors

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3365
2016-06-29 19:06:09 +00:00
Lars K.W. Gohlke
af0bc95de5 lib/model: Refactor encapsulation of the folder scanning
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3017
2016-06-29 06:37:34 +00:00
Jakob Borg
4bf3e7485b lib/events: Make events test less likely to fail 2016-06-28 08:29:49 +02:00
Antony Male
7ef2743964 lib/events: Introduce per-subscription event IDs (fixes #3335)
Events API consumers rely on being able to detect that events were skipped
by the fact that the event ID has increased by more than 1. This is
documented, and is absolutely necessary when trying to maintain a local
model of Syncthing's state.

With the introduction of LocalChangeDetected, which is not exposed to the
Events API, this contract was broken.

This commit introduces separate concepts of a "Global ID" and a
"Subscription ID". The Global ID of an event is unique across all
subscriptions. The Subscription ID is local to a particular subscription,
and always increments by 1. They are both exposed over the Events API, but
the Subscription ID uses the key "id" for backwards compatibility, and
the "?since=xx" parameter refers to the Subscription ID (making the Global
ID for information only).

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3351
LGTM: calmh
2016-06-27 21:18:58 +00:00
Jakob Borg
a165838cbd lib/model: Decrease max temp filename length (fixes #3338, fixes #3355)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3356
2016-06-27 11:47:40 +00:00
Jakob Borg
9d16f4545d lib/events: Add logging/receiving benchmark 2016-06-27 10:26:59 +02:00
Jakob Borg
d57e6808cc lib/db: Fix alignment crash on 32 bit platforms
Fixes #3347
Fixes #3348
Fixes #3349

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3350
2016-06-26 13:40:51 +00:00
Jakob Borg
ac3b03881a gui: Add addresses for disconnected devices (fixes #3340)
Also fixes an issue where the discovery cache call would only return the
newest cache entry for a given device instead of the merged addresses
from all cache entries (which is more useful).

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3344
2016-06-26 10:47:23 +00:00
Jakob Borg
b0d03d1f1c lib/config: Retain slash at end of path after expanding ~ (fixes #2782)
The various path cleaning operations done in in cleanedPath() removes
it, so we make sure it's added again at the end. This makes adding the
slash in prepare() unnecessary, but keep it anyway for display purposes
(people looking at the config).

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3342
2016-06-26 10:17:20 +00:00
Jakob Borg
a2dcffcca2 lib/nat: Avoid concurrent reset of NAT timer (fixes #3337)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3341
2016-06-26 10:17:12 +00:00
Jakob Borg
9323f0faf8 lib/model: Refactor CheckFolderHealth into separate methods
While attempting to fix #2782 I thought the problem was the
CheckFolderHealth method, so I cleaned it up. That turned out not to be
the case, but I think this is better anyhow.

It also moves the "create folder and marker if the folder was empty in
the index" code to StartFolder where I think it makes better sense.

This is covered by a number of existing tests.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3343
2016-06-26 10:07:27 +00:00
Jakob Borg
f343c8ba36 lib/model, lib/scanner: Silence vet warnings
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3333
2016-06-20 21:00:39 +00:00
Audrius Butkevicius
502bee9a09 lib/osutil: Return "/" as filesystem root on non-windows (fixes #3321)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3332
2016-06-20 20:25:00 +00:00
Jakob Borg
b779e22205 lib/model: Don't set ignore bit when it's already set
This adds a metric for "committed items" to the database instance that I
use in the test code, and a couple of tests that ensure that scans that
don't change anything also don't commit anything.

There was a case in the scanner where we set the invalid bit on files
that are ignored, even though they were already ignored and had the
invalid bit set. I had assumed this would result in an extra database
commit, but it was in fact filtered out by the Set... Anyway, I think we
can save some work on not pushing that change to the Set at all.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3298
2016-06-13 17:44:03 +00:00
Jakob Borg
6f5ca53f99 lib/connections: Limit rate at which we print warnings about version mismatch
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3291
2016-06-09 12:30:35 +00:00
Jakob Borg
d507126101 lib/protocol: Understand older/newer Hello messages (fixes #3287)
This is in preparation for future changes, but also improves the
handling when talking to pre-v0.13 clients. It breaks out the Hello
message and magic from the rest of the protocol implementation, with the
intention that this small part of the protocol will survive future
changes.

To enable this, and future testing, the new ExchangeHello function takes
an interface that can be implemented by future Hello versions and
returns a version indendent result type. It correctly detects pre-v0.13
protocols and returns a "too old" error message which gets logged to the
user at warning level:

   [I6KAH] 09:21:36 WARNING: Connecting to [...]:
     the remote device speaks an older version of the protocol (v0.12) not
     compatible with this version

Conversely, something entirely unknown will generate:

   [I6KAH] 09:40:27 WARNING: Connecting to [...]:
     the remote device speaks an unknown (newer?) version of the protocol

The intention is that in future iterations the Hello exchange will
succeed on at least one side and ExchangeHello will return the actual
data from the Hello together with ErrTooOld and an even more precise
message can be generated.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3289
2016-06-09 10:50:14 +00:00
perewa
11b9212948 cmd/syncthing: Increase timeout in hello message exchange
Required to establish connections on high latency links

Skip-check: authors

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3286
2016-06-08 19:46:54 +00:00
scienmind
bc794e7c15 lib/connections: Relay failures should be informative, not warning
Skip-check: authors

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3263
2016-06-04 10:41:36 +00:00
Jakob Borg
6a67921e40 vendor: Revert to github.com/jackpal/gateway instead of fork
It now includes all the fixes

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3246
2016-06-02 20:40:30 +00:00
Majed Abdulaziz
48245effdf lib/model, lib/stats: Keep track of folder's last scan time (ref #3143)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3250
LGTM: calmh, AudriusButkevicius
2016-06-02 19:26:52 +00:00
Audrius Butkevicius
6f63909c65 lib/db,cmd/stindex: Expose VersionList and use it in stindex
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3231
2016-05-31 19:29:26 +00:00
Audrius Butkevicius
1612baca92 gui: /rest/system/browse with no arguments returns drives on Windows (ref #3201)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3203
2016-05-31 19:27:07 +00:00
Jakob Borg
4970bd7f65 lib/relay: Correctly get IP from remote addr via proxy (fixes #3223)
Correctly handles addresses, and fixes one more panicing place.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3230
2016-05-31 14:42:10 +00:00
Jakob Borg
ac40b27c79 lib/connections: Handle wrapped connection in SetTCPOptions (fixes #3223)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3225
2016-05-31 08:11:57 +00:00
Jakob Borg
4453236949 vendor: Update github.com/gobwas/glob (fixes #3174)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3207
2016-05-28 04:43:54 +00:00
Audrius Butkevicius
c2dc4a8e06 lib/db: Have prefix should be normalized
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3206
2016-05-28 04:18:31 +00:00
Audrius Butkevicius
92a23da3ec lib/model: Make the (?d) prefix actually work
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3205
2016-05-28 04:17:34 +00:00
Audrius Butkevicius
242db26343 lib/osutil: Fix globbing at root (fixes #3201)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3202
2016-05-28 04:13:34 +00:00
Audrius Butkevicius
87701339fe lib/nat, lib/connections: Fix a few issues with NAT traversal
1. For the same internal port we ask for the same external port on all devices. This can be a problem if one device speaks over two protocols.
2. Always add a nil address even if we managed to get external address of the gateway, just because the gateway might be in DMZ behind another gateway.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3196
2016-05-27 06:28:46 +00:00
Jakob Borg
9bb5988b4e lib/model: Don't deadlock when returning temp index block counts
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3194
2016-05-26 09:16:08 +00:00