This adds our short device ID to the basic auth realm. This has at least
two consequences:
- It is different from what's presented by another device on the same
address (e.g., if I use SSH forwards to different dives on the same
local address), preventing credentials for one from being sent to
another.
- It is different from what we did previously, meaning we avoid cached
credentials from old versions interfering with the new login flow.
I don't *think* there should be things that depend on our precise realm
string, so this shouldn't break any existing setups...
Sneakily this also changes the session cookie and CSRF name, because I
think `id.Short().String()` is nicer than `id.String()[:5]` and the
short ID is two characters longer. That's also not a problem...
This makes the new default $XDG_STATE_HOME/syncthing or
~/.local/state/syncthing, while still looking in legacy locations first
for existing installs.
Note that this does not *move* existing installs, and nor should we.
Existing paths will continue to be used as-is, but the user can move the
dir into the new place if they want to use it (as they could prior to
this change as well, for that matter).
### Documentation
Needs update to the config docs about our default locations.
lib/fs: Fix conflicts on Android due to fluctuating inode change time
[1] added inode change time to file info in order to support syncing
extended attributes. However, in the case of Android, this inode change
time fluctuates, leading to unexpected conflicts even when the user has
not even touched the files on the Android device itself. Thus, in order
to prevent those conflicts from happening, do not write inode change
time on Android.
[1] 6cac308bcd
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
### Purpose
Deduplicated files are apparently considered 'irregular' under the hood,
this causes them to simply be ignored by Syncthing. This change is more
of a workaround than a proper fix, as the fix should probably happen in
the underlying libraries? - which may take some time. In the meanwhile,
this change should make deduplicated files be treated as regular files
and be indexed and synced as they should.
### Testing
Create some volume where deduplication is turned on (see the relevant
issue for details, including a proper description of how to reproduce
it). Prior to this change, the deduplicated files were simply ignored
(even by the indexer). After this change, the deduplicated files are
being index and synced properly.
This is motivated by the Android app:
https://github.com/syncthing/syncthing-android/pull/1982#issuecomment-1752042554
The planned fix in response to basic auth behaviour changing in #8757
was to add the `Authorization` header when opening the WebView, but it
turns out the function used only applies the header to the initial page
load, not any subsequent script loads or AJAX calls. The
`basicAuthAndSessionMiddleware` checks for no-auth exceptions before
checking the `Authorization` header, so the header has no effect on the
initial page load since the `/` path is a no-auth exception. Thus the
Android app fails to log in when opening the WebView.
This changes the order of checks in `basicAuthAndSessionMiddleware` so
that the `Authorization` header is always checked if present, and a
session cookie is set if it is valid. Only after that does the
middleware fall back to checking for no-auth exceptions.
`api_test.go` has been expanded with additional checks:
- Check that a session cookie is set whenever correct basic auth is
provided.
- Check that a session cookie is not set when basic auth is incorrect.
- Check that a session cookie is not set when authenticating with an API
token (either via `X-Api-Key` or `Authorization: Bearer`).
And an additional test case:
- Check that requests to `/` always succeed, but receive a session
cookie when correct basic auth is provided.
I have manually verified that
- The new assertions fail if the `createSession` call is removed in
`basicAuthAndSessionMiddleware`.
- The new test cases in e6e4df4d7034302b729ada6d91cff6e2b29678da fail
before the change in 0e47d37e738d4c15736c496e01cd949afb372e71 is
applied.
improve parsing of gui-address overrides
make checks for whether the gui-address is overridden consistent by
checking whether the environment variable is set and not an empty
string. the `Network()` function however checked for the inclusion of
a slash instead of the presence of any characters. If the config file's
gui address was set to a unix socket and the gui override to a tcp
address, then the function would have wrongly returned "unix".
the `URL()` function always returned the config file's gui address if a
unix socket was configured, even if an override was specified.
the `URL()` function wrongly formatted unix addresses. the http(s)
protocol was used as the sheme and the path was percent escaped. because
of the previous bug, this could only be triggered if the config file's
gui address was tcp and an unix socket override was given.
simplify the `useTLS()` function's codepath for overrides.
Co-authored-by: digital <didev@dinid.net>
I don't really understand under what circumstances, but sometimes these
calls panic with a "panic: counter cannot decrease in value" because the
value passed to Add() was negative.
In principle a connection can close while it's in progress with
starting, and then it's undefined if we wait for goroutines to exit etc.
With this change, we will wait for start to complete before starting to
stop everything.
This adds the ability to have multiple concurrent connections to a single device. This is primarily useful when the network has multiple physical links for aggregated bandwidth. A single connection will never see a higher rate than a single link can give, but multiple connections are load-balanced over multiple links.
It is also incidentally useful for older multi-core CPUs, where bandwidth could be limited by the TLS performance of a single CPU core -- using multiple connections achieves concurrency in the required crypto calculations...
Co-authored-by: Simon Frei <freisim93@gmail.com>
Co-authored-by: tomasz1986 <twilczynski@naver.com>
Co-authored-by: bt90 <btom1990@googlemail.com>
Instead of separately tracking the token.
Also changes serviceMap to have a channel version of RemoveAndWait, so
that it's possible to do the removal under a lock but wait outside of
the lock. And changed where we do that in connection close, reversing
the change that happened when I added the serviceMap in 40b3b9ad1.
By creating the http.Transport and tls.Configuration ourselves we
override some default behavior and end up with a client that speaks only
HTTP/1.1.
This adds a call to http.ConfigureTransport to do the relevant magic to
enable HTTP/2.
Also tweaks the keepalive settings to be a little kinder to the
server(s).
The allowed IPv4 ranges are the same as before. But we now also accept IPv6 addresses in the ULA range FC00::/7. These addresses don't require an interface identifier and are roughly equivalent to the IPv4 private ranges.
Typical usecases:
VPN interface IPs: Wireguard, OpenVPN, Tailscale, ...
fixed IPv6 LAN addressing while the provider assigns a dynamic prefix. e.g used by pihole
https://cs.opensource.google/go/go/+/refs/tags/go1.21.0:src/net/ip.go;l=146
* lib/versioner: Factor out DefaultPath constant.
Replace several instances where .stversions is named literally to all
use the same definition in the versioner package. Exceptions are the
packages where a cyclic dependency on versioner is impossible, or some
tests which combine the versions base path with other components.
* lib/versioner: Fix comment about trash can in simple versioner.
* lib/versioner: Fix wrong versioning type string in error message.
The error message shows the folder type instead of the versioning
type, although the correct field is used in the comparison.
Safety check added in v1.23.6 introduced bug. Bug unshares folders with untrusted devices if folder does not have an encryption password set, regardless of whether the folder is shared with the untrusted device as encrypted or not. Prevents sharing with untrusted devices in some cases where sharing would be encrypted.
Patch preserves safety check but permits sharing folders with untrusted devices if they are shared as encrypted.
Signed-off-by: kewiha <keithh@protonmail.com>
Currently, historically, we look for the `X-API-Key` header to
authenticate with an API key. There's nothing wrong with this, but in
some scenarios it's easier to produce an `Authorization` header with a
`Bearer $token` content, which is nowadays more common. This change adds
support for both, so that we will accept an API key either in our custom
header or as a bearer token.
Currently, because of devices with unset RTC clock, the 100% percentile
for Uptime on [1] is calculated since the Unix epoch which is useless as
far as usage statistics are concerned. Thus, if the Syncthing start time
is set to a past date, assume that the clock is wrong and do not even
try to report the uptime.
[1] https://data.syncthing.net
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
Co-authored-by: Jakob Borg <jakob@kastelo.net>
refactor: replace empty slice literal with `var`
An empty slice can be represented by `nil` or an empty slice literal. They are
functionally equivalent — their `len` and `cap` are both zero — but the `nil`
slice is the preferred style. For more information about empty slices,
see [Declaring Empty Slices](https://github.com/golang/go/wiki/CodeReviewComments#declaring-empty-slices).
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>
refactor: fix unused method receiver
Methods with unused receivers can be a symptom of unfinished refactoring or a bug. To keep
the same method signature, omit the receiver name or '_' as it is unused.
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>
refactor: unused parameter should be replaced by underscore
Unused parameters in functions or methods should be replaced with `_`
(underscore) or removed.
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>
This adds an environment variable STVERSIONEXTRA that, when set, gets
added to the version information in the API and GUI.
The purpose of all this is to be able to communicate something about the
bundling or packaging, through the log & GUI and the end user, to the
potential person supporting it -- i.e., us. :) A wrapper can set this
variable to indicate that Syncthing is being run via `SyncTrayzor`,
`Syncthing-macOS`, etc., and thus indicate to the end user that the GUI
they are looking at is perhaps not the only source of truth and
management for this instance.
With this change, error messages include the offending characters or
name parts. Examples:
nul.txt: name is invalid, contains Windows reserved name: "nul"
foo>bar.txt: name is invalid, contains Windows reserved character: ">"
foo \bar.txt: name is invalid, must not end in space or period on Windows
This prevents combining untrusted with introducer and auto-accept, and
also verifies that folders shared with untrusted devices have passwords
at config loading time.
Co-authored-by: Simon Frei <freisim93@gmail.com>
We usually want to ensure that our own device is present. However if the
given device ID is the empty ID, we shouldn't do that. This is a
legimate (though way too non-obvious) use-case when opening the config
without knowing/caring about the device ID.
* Platform data (ownership, xattrs, etc.) is now set correctly for newly-received folders, even if the received folder has the NoPermissions flag.
* Call setPlatformData on receivers that have ignorePerms set to true.
This fixes various test issues with Go 1.20.
- Most tests rewritten to use fakefs where possible
- Some tests that were already skipped, or dubious (invasive,
unmaintainable, unclear what they even tested) have been removed
- Some actual code rewritten to better support testing in fakefs
Co-authored-by: Eric P <eric@kastelo.net>
In the sequence of loading ignores, the error File Does Not Exist is not being considered a fatal error, since the .stignore file is allowed to not exist. However, included ignore files also tossed that same error in case those do not exist while in those cases it's considered an error and it should lead to the folder stopping. Changing the error when opening an included ignore file to something other than the regular does fix this issue, as in it now works again as described in the Documentation.
This makes the various protocol priorities configurable among the other
options. With this, it's possible to prefer QUIC over TCP for WAN
connections, for example. Both sides need to be similarly configured for
this to work properly.
The default priority order remains the same as previously (TCP, QUIC,
Relay, with LAN better than WAN).
To make this happen I made each dialer & listener more priority aware,
and moved the check for whether a connection is LAN or not into the
dialer / listener -- this is the new "lanChecker" type that's passed
around.
In the original fix in #8563 I simply forgot this. Which meant #8556
wasn't actually fixed, as the trialer size would have been 0 (default),
and thus we would have still sent the inflated size to encrypted peers.
lib/model: Fix file size inconsisency due to enc. trailer
Fixes a regression due to PR #8563, while arguable the bug was actually
introduced in a much older PR #7155, but didn't have any bad effects so
far:
We account for the encryption trailer in the db updater routine,
calculating the file-info size there. However there's no guarantee that
the file-info at this point is still the exact same as when it was
written. It was before, but isn't anymore since introducing the new
EncryptedTrailerSize field.
Fix: Adjust the size in the info at the same place where the trailer is
written, i.e. we definitely have the actual size on disk.
The layout of the request differs based on whether it comes from an
untrusted device or a trusted device with encrypted enabled. Handle
both.
Closes#8819.
Allow the watcher delay to take fractional values, effectively allowing
for much shorter delays. The minimum value is limited at 0.01, which
effectively translates to 10ms. This is required in order to guarantee
that there is still enough time to aggregate multiple single change
events.
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
This adds a cache to the expensive key generation operations. It's fixes
size LRU/MRU stuff to keep memory usage bounded under absurd conditions.
Also closes#8600.
This adds the BlocksHash field from the FileInfo to our API output. It
can be useful for debugging, or for external tools. I'm intentionally
leaving it as an opaque base64 string because no meaning should be
derived from it: it's just a string.
This makes sure the service manager doesn't interpret timeout errors, or any other error, as a signal to stop the service instead of restarting it.
I added it directly to our service utility function, as it may help catch other instances of the same problem... We would typically want timeouts etc to be a retryable error, unless it is the top level context that has timed out and we check for that specifically.
This adds a word to the version string when running containerized. The
purpose is mostly to facilitate troubleshooting via screenshot by
"leaking" this rather important aspect of the setup. Additionally, the
version row gets "no-overflow-ellipsis" treatment so that the whole
thing is actually visible in the GUI and the (now useless) tooltip is
removed. In production releases this won't make a difference as the
whole thing will typically fit, but in odd setups it provides more info
up front.
* lib/connections: Cache isLAN decision for later external access.
The check whether a remote device's address is on a local network
currently happens when handling the Hello message, to configure the
limiters. Save the result to the ConnectionInfo and pass it out as
part of the model's ConnectionInfo struct in ConnectionStats().
* gui: Use provided connection attribute to distinguish LAN / WAN.
Replace the dumb IP address check which didn't catch common cases and
actually could contradict what the backend decided. That could have
been confusing if the GUI says WAN, but the limiter is not actually
applied because the backend thinks it's a LAN.
Add strings for QUIC and relay connections to also differentiate
between LAN and WAN.
* gui: Redefine reception level icons for all connection types.
Move the mapping to the JS code, as it is much easier to handle
multiple switch cases by fall-through there.
QUIC is regarded no less than TCP anymore. LAN and WAN make the
difference between levels 4 / 3 and 2 / 1:
{TCP,QUIC} LAN --> {TCP,QUIC} WAN --> Relay LAN --> Relay WAN -->
Disconnected.
Previous debug input didn't really give enough info to show what was
happening, while it also printed full block lists which are enormously
verbose. Now it consistently prints 1. what it sees on disk, 2. what it
got from CurrentFile (without blocks), 3. the action taken on that file.
There are some situations where an upgrade wouldn't be supported, even though the noUpgrade bool isn't set. So when handling the errors that are caused by this, when attempting an upgrade, it shouldn't lead to some sort of offline-message/restart/warning/etc...
I added some checks on specific errors related to this and return a 501 (Not Implemented) response instead, in case of an "UpgradeUnsupported"-error. Additionally, on the GUI-side, the 501-response is now not to be considered an error to act upon.
* implement authentication via token for relaysrv
Make replaysrv check for a token before allowing clients to
join. The token can be set via the replay-uri.
* fix formatting
* key composite literal
* do not error out if auth material is provided but not needed
* remove unused method receiver
* clean up unused parameter in functions
* cleaner token handling, disable joining the pool if token is set.
* Keep backwards compatibility with older clients.
In prior versions of the protocol JoinRelayRequest did not have a
token field. Trying to unmarshal such a request will result in
an error. Return an empty JoinRelayRequest, that is a request
without token, instead.
Co-authored-by: entity0xfe <entity0xfe@my.domain>
The restore function of Trash Can ran a rename at the end regardless of whether there was anything to rename. In this case, when the file-to-be-restored did not exist in the destination folder, this resulted in an error. I added a simple check, keeping track of whether the file existed prior to restoring it in the destination folder and depending on this value it will now return nil after the restoration to prevent the renaming function to kick off. Added a test for this specific edge-case as well.
This adds support for syncing extended attributes on supported
filesystem on Linux, macOS, FreeBSD and NetBSD. Windows is currently
excluded because the APIs seem onerous and annoying and frankly the uses
cases seem few and far between. On Unixes this also covers ACLs as those
are stored as extended attributes.
Similar to ownership syncing this will optional & opt-in, which two
settings controlling the main behavior: one to "sync" xattrs (read &
write) and another one to "scan" xattrs (only read them so other devices
can "sync" them, but not apply any locally).
Co-authored-by: Tomasz Wilczyński <twilczynski@naver.com>
The cleaning logic in util.go was used by Simple and Trashcan but only
really suited Trashcan since it works based on mtimes which Simple does
not use. The cleaning logic in util.go was moved to trashcan.go.
Staggered and Simple seemed to be able to benefit from the same base so
util.go now has the base for those two with an added parameter which
takes a function so it can still handle versioner-specific logic to
decide which files to clean up. Simple now also correctly cleans files
based on their time-stamp in the title together with a specific maximum
amount to keep. The Archive function in Simple.go was changed to get rid
of duplicated code.
Additionally the trashcan testcase which was used by Trashcan as well as
Simple was moved from versioner_test.go to trashcan_test.go to keep it
clean, there was no need to keep it in a separate test file
This replaces old style errors.Wrap with modern fmt.Errorf and removes
the (direct) dependency on github.com/pkg/errors. A couple of cases are
adjusted by hand as previously errors.Wrap(nil, ...) would return nil,
which is not what fmt.Errorf does.
Locally paused folders will fail on checkFolderRunningLocked() and
therefore abort the loop. Avoid this by skipping paused folders
directly.
Co-authored-by: Jakob Borg <jakob@kastelo.net>
* lib/locations: Fix enum values camelCase.
* lib/locations: Remove unused FailuresFile.
* cmd/syncthing: Turn around role of locations storage.
Previously the locations package was used to provide default paths,
possibly with an overridden home directory. Extra paths supplied on
the command line were handled and passed around in the options object.
To make the changed paths available to any other interested package,
override the location setting from the option if supplied, instead of
vice versa when not supplied. Adapt code using this to read from the
locations package instead of passing through the options object.
* lib/locations: Refactor showPaths to locations package.
Generate a reusable string in locations.PrettyPrintPaths().
Enumerating all possible locations in different packages is error
prone, so add a new public function to generate the listing as a
string in the locations package. Adapt cmd/syncthing --paths to use
that instead of its own console output.
* lib/locations: Include CSRF token in pretty printed paths.
* lib/api: New endpoint /rest/system/paths.
The paths should be available for troubleshooting from a running
instance. Using the --paths CLI option is not easy in some
environments, so expose the locations mapping to a JSON endpoint.
Add utility function ListExpandedPaths() that also filters out any
entries which still contain variable placeholders.
* gui: List runtime paths in separate log viewer tab.
* Wrap paths.
* lib/syncthing: Utilize locations.Get() instead of passing an arg.
* Include base directories, move label to table caption.
* gui: Switch to hard-coded paths instead of iterating over all.
* gui: Break aboutModalView into tabs.
Use tabs to separate authors from included third-party software.
* gui: Move paths from log viewer to about modal.
* lib/locations: Adjust pretty print output order to match GUI.
* gui, authors: Remove additional bot names and fix indent.
The indentation changed because of the tabbed about dialog, fix the
authors script to respect that.
Skip Syncthing*Automation in authors list as well.
* Update AUTHORS list to remove bot names.
* Revert AUTHORS email order change.
* Do not emphasize DB and log file locations.
* Review line wrapping.
* review part 1: strings.Builder, naming
* Rename and extend locations.Set() with error handling.
Remodel the Override() function along the existing SetBaseDir() and
rename it to simply Set(). Make sure to use absolute paths when given
log file or GUI assets override options. Add proper error reporting
if that goes wrong.
* Remove obsolete comment about empty logfile option.
* Don't filter out unexpanded baseDir placeholders, only ${timestamp}.
* Restore behavior regarding special "-" logfile argument.
If the option is given, but with empty value, assume the no log
file (same as "-"). Don't try to convert the special value to an
absolute path though and document this fact in a comment for the Set()
function.
* Use template to check for location key validity.
* Don't filter out timestamp placeholders.
* lib/api: Remove paths from /rest/system/status.
* lib/ur: Properly initialize map in failure data (fixes#8479)
Co-authored-by: Jakob Borg <jakob@kastelo.net>
Also adds idle time and keepalive parameters because how this is
configured has changed in the new package version. The values are those
that seems like might already be default, if keep-alives were enabled,
which is not obvious from the doc comments.
Also, Go 1.19 gofmt reformatting of comments.
all: Add package runtimeos for runtime.GOOS comparisons
I grew tired of hand written string comparisons. This adds generated
constants for the GOOS values, and predefined Is$OS constants that can
be iffed on. In a couple of places I rewrote trivial switch:es to if:s,
and added Illumos where we checked for Solaris (because they are
effectively the same, and if we're going to target one of them that
would be Illumos...).
This adds support for syncing ownership on Unixes and on Windows. The
scanner always picks up ownership information, but it is not applied
unless the new folder option "Sync Ownership" is set.
Ownership data is stored in a new FileInfo field called "platform data". This
is intended to hold further platform-specific data in the future
(specifically, extended attributes), which is why the whole design is a
bit overkill for just ownership.
* lib/model: Override scan config for auto-accepted encrypted folders.
Encrypted folders should not have the fs watcher enabled and rarely
benefit from a scheduled rescan. The GUI adjusts the suggested
settings (watcher disabled, one day rescan interval) when accepting a
receive-encrypted folder. Mirror that behavior to the auto-accept
case where the GUI is not involved.
Versioning also does not work well for encrypted folders, same
treatment.
