Otherwise if the file grows during scanning the block list will be out
of sync with the stated size and things get confused. We could fixup the
size afterwards based on the block list, but then we might see other
inconsistencies as the mtime should have changed to reflect the new size
etc. Better stick to the original state and let the next scan pick up
the change.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3442
We could have a file to sync with permissions rw------- but we'd create
the temp file with rw-rw-rw- minus umask, usually rw-r--r--. This
potentially exposes private data while the file is being synced.
Similarly, when ignorePerms was set and we were reusing a temp files we
would set the permissions to rw-r--r-- explicitly, potentially
overriding a strict umask that would otherwise have had the file be
rw-------.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3437
The previous commit loosened the locking around database updates.
Apparently that was not fine - what happens is that parallell updates
to the same file for different devices stomp on each others updates to
the global index, leaving it missing one of the two devices.
This lets us add message types in the future, for authentication or
other purposes, without completely breaking old clients. I see this as
similar behavior to adding fields to messages - newer clients must
simple be aware that older ones may ignore the message and act
accordingly.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3390
This slightly changes the interface used for committing configuration
changes. The two parts are now:
- VerifyConfiguration, which runs synchronously and locked, and can
abort the config change. These callbacks shouldn't *do* anything
apart from looking at the config changes and saying yes or no. No
change from previously.
- CommitConfiguration, which runs asynchronously (one goroutine per
call) *after* replacing the config and releasing any locks. Returning
false from these methods sets the "requires restart" flag, which now
lives in the config.Wrapper.
This should be deadlock free as the CommitConfiguration calls can take
as long as they like and can wait for locks to be released when they
need to tweak things. I think this should be safe compared to before as
the CommitConfiguration calls were always made from a random background
goroutine (typically one from the HTTP server), so it was always
concurrent with everything else anyway.
Hence the CommitResponse type is gone, instead you get an error back on
verification failure only, and need to explicitly check
w.RequiresRestart() afterwards if you care.
As an added bonus this fixes a bug where we would reset the "requires
restart" indicator if a config that did not require restart was saved,
even if we already were in the requires-restart state.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3386
A random "instance ID" is generated on each start of the local discovery
service. The instance ID is included in the announcement. When we see a
new instance ID we treat is a new device and respond with an
announcement of our own. Hence devices get to know each other quickly on
restart.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3385
This changes the BEP protocol to use protocol buffer serialization
instead of XDR, and therefore also the database format. The local
discovery protocol is also updated to be protocol buffer format.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3276
LGTM: AudriusButkevicius
This is a supplement patch to commit a58f69b which only fixed global
discovery. This patch adds the missing parts for the local discovery.
If the listen address scheme is set to tcp4:// or tcp6:// and no
explicit host is specified, an address should not be considered if the
source address does not match this scheme.
This prevents invalid URIs like tcp4://<IPv6 address>:<port> or tcp6://<IPv4
address>:<port> for local discovery.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3380
This contains the following behavioral changes:
- Duplicate folder IDs is now fatal during startup
- Invalid folder flags in the ClusterConfig is fatal for the connection
(this will go away soon with the proto changes, as we won't have any
unknown flags any more then)
- Empty path is a folder error reported at runtime
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3370
Events API consumers rely on being able to detect that events were skipped
by the fact that the event ID has increased by more than 1. This is
documented, and is absolutely necessary when trying to maintain a local
model of Syncthing's state.
With the introduction of LocalChangeDetected, which is not exposed to the
Events API, this contract was broken.
This commit introduces separate concepts of a "Global ID" and a
"Subscription ID". The Global ID of an event is unique across all
subscriptions. The Subscription ID is local to a particular subscription,
and always increments by 1. They are both exposed over the Events API, but
the Subscription ID uses the key "id" for backwards compatibility, and
the "?since=xx" parameter refers to the Subscription ID (making the Global
ID for information only).
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3351
LGTM: calmh
Also fixes an issue where the discovery cache call would only return the
newest cache entry for a given device instead of the merged addresses
from all cache entries (which is more useful).
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3344
The various path cleaning operations done in in cleanedPath() removes
it, so we make sure it's added again at the end. This makes adding the
slash in prepare() unnecessary, but keep it anyway for display purposes
(people looking at the config).
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3342
While attempting to fix#2782 I thought the problem was the
CheckFolderHealth method, so I cleaned it up. That turned out not to be
the case, but I think this is better anyhow.
It also moves the "create folder and marker if the folder was empty in
the index" code to StartFolder where I think it makes better sense.
This is covered by a number of existing tests.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3343
This adds a metric for "committed items" to the database instance that I
use in the test code, and a couple of tests that ensure that scans that
don't change anything also don't commit anything.
There was a case in the scanner where we set the invalid bit on files
that are ignored, even though they were already ignored and had the
invalid bit set. I had assumed this would result in an extra database
commit, but it was in fact filtered out by the Set... Anyway, I think we
can save some work on not pushing that change to the Set at all.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3298
This is in preparation for future changes, but also improves the
handling when talking to pre-v0.13 clients. It breaks out the Hello
message and magic from the rest of the protocol implementation, with the
intention that this small part of the protocol will survive future
changes.
To enable this, and future testing, the new ExchangeHello function takes
an interface that can be implemented by future Hello versions and
returns a version indendent result type. It correctly detects pre-v0.13
protocols and returns a "too old" error message which gets logged to the
user at warning level:
[I6KAH] 09:21:36 WARNING: Connecting to [...]:
the remote device speaks an older version of the protocol (v0.12) not
compatible with this version
Conversely, something entirely unknown will generate:
[I6KAH] 09:40:27 WARNING: Connecting to [...]:
the remote device speaks an unknown (newer?) version of the protocol
The intention is that in future iterations the Hello exchange will
succeed on at least one side and ExchangeHello will return the actual
data from the Hello together with ErrTooOld and an even more precise
message can be generated.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3289
1. For the same internal port we ask for the same external port on all devices. This can be a problem if one device speaks over two protocols.
2. Always add a nil address even if we managed to get external address of the gateway, just because the gateway might be in DMZ behind another gateway.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3196
The intention for this package is to provide a combination of the
security of crypto/rand and the convenience of math/rand. It should be
the first choice of random data unless ultimate performance is required
and the usage is provably irrelevant from a security standpoint.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3186
The math/rand package contains lots of convenient functions, for example
to get an integer in a specified range without running into issues
caused by just truncating a number from a different distribution and so
on. But it's insecure, and we use if for things that benefit from being
more secure like session IDs, CSRF tokens and API keys.
This implements a math/rand.Source that reads from crypto/rand.Reader,
this bridging the gap between them. It also updates our RandomString to
use the new source, thus giving us secure session IDs and CSRF tokens.
Some future work remains:
- Fix API keys by making the generation in the UI use this code as well
- Refactor out these things into an actual random package, and audit
our use of randomness everywhere
I'll leave both of those for the future in order to not muddy the waters
on this diff...
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3180
Switch to my forked version which contains a fix for this issue. I'll
track upstream in the future if things update there, and attempt to
contribute back fixes...
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3149
Without this the summary service doesn't know to recalculate completion
percentage for remote devices when DownloadProgress messages come in.
That means that completion percentage isn't updated in the GUI while
transfers of large files are ongoing. With this change, it updates
correctly.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3144
I think this better reflects what it means. Also tweaks the verbose
format to be more like our other things and lightly refactors the code
to not have the boolean and include the folder in the event.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3121
This was fixed upstream due to our ticket, so we no longer need the
manual handling of commas. Keep the tests and better debug output around
though.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3081
The old usage pattern was to create a Walker with a bunch of attributes,
then call Walk() on it and nothing else. This extracts the attributes
into a Config struct and exposes a Walk(cfg Config) method instead, as
there was no reason to expose the state-holding walker type.
Also creates a few no-op implementations of the necessary interfaces
so that we can skip nil checks and simiplify things here and there.
Definitely look at this diff without whitespace.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3060
Just an optimization. Required exposing the priority from the factory,
so made that an interface with an extra method instead of just a func
type.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3071
This fixes the deadlock by reducing where we hold the various locks. To
start with it splits up the existing "mut" into a "listenersMut" and a
"curConMut" as these are the two things being protected and I can see no
relation between them that requires a shared lock. It also moves all
model calls outside of the lock, as I see no reason to hold the lock
while calling the model (and it's risky, as proven).
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3069
When doing prefix scans in the database, "foo" should not be considered
a prefix of "foo2". Instead, it should match "foo" exactly and also
strings with the prefix "foo/". This is more restrictive than what the
standard leveldb prefix scan does so we add some code to enforce it.
Also exposes the initialScanCompleted on the rwfolder for testing, and
change it to be a channel (so we can wait for it from another
goroutine). Otherwise we can't be sure when the initial scan has
completed, and we need to wait for that or it might pick up changes
we're doing at an unexpected time.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3067
The VersioningConfig change is because it defaults to nil but gets
deserialized to map[string]string{}. Now prepare() enforces a single
representation of the empty map.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3065
Because json.NewDecoder(r).Decode(&v) doesn't necessarily consume all
data on the reader, that means an HTTP connection can't be reused. We
don't do a lot of HTTP traffic where we read JSON responses, but the
discovery is one such place. The other two are for POSTs from the GUI,
where it's not exactly critical but still nice if the connection still
can be keep-alive'd after the request as well.
Also ensure that we call req.Body.Close() for clarity, even though this
should by all accounts not really be necessary.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3050
New signature is the HMAC of archive name (which includes the release
version and architecture) plus the contents of the binary. This is
expected in a new file "release.sig" which may be present in a
subdirectory. The new release tools put this in [.]metadata/release.sig.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3043
1. Removes separate relay lists and relay clients/services, just makes it a listen address
2. Easier plugging-in of other transports
3. Allows "hot" disabling and enabling NAT services
4. Allows "hot" listen address changes
5. Changes listen address list with a preferable "default" value just like for discovery
6. Debounces global discovery announcements as external addresses change (which it might alot upon starting)
7. Stops this whole "pick other peers relay by latency". This information is no longer available,
but I don't think it matters as most of the time other peer only has one relay.
8. Rename ListenAddress to ListenAddresses, as well as in javascript land.
9. Stop serializing deprecated values to JSON
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/2982
