Commit Graph

2202 Commits

Author SHA1 Message Date
DerRockWolf
debbe726e0
lib/connections: Add syncthing_connections_active metric (fixes #9527) (#9528)
### Purpose

Adds a new metric `syncthing_connections_active` which equals to the
amount of active connections per device.

Fixes #9527 

<!--
Describe the purpose of this change. If there is an existing issue that
is
resolved by this pull request, ensure that the commit subject is on the
form
`Some short description (fixes #1234)` where 1234 is the issue number.
-->

### Testing

I've manually tested it by running syncthing with these changes locally
and examining the returned metrics from `/metrics`.
I've done the following things:
- Connect & disconnect a device
- Increase & decrease the number of connections and verify that the
value of the metric matches with the amount displayed in the GUI.

### Documentation

https://github.com/syncthing/docs/blob/main/includes/metrics-list.rst
needs to be regenerated with
[find-metrics.go](https://github.com/syncthing/docs/blob/main/_script/find-metrics/find-metrics.go)

## Authorship

Your name and email will be added automatically to the AUTHORS file
based on the commit metadata.

---------

Co-authored-by: Jakob Borg <jakob@kastelo.net>
2024-05-04 22:31:37 +02:00
Jakob Borg
79ae24df76
lib/nat: Don't crash on empty address list (fixes #9503) (#9504) 2024-04-11 13:23:29 +02:00
Jakob Borg
61b94b9ea5
lib/db: Drop indexes for outgoing data to force refresh (ref #9496) (#9502)
### Purpose

Resend our indexes since we fixed that index-sending issue.

I made a new thing to only drop the non-local-device index IDs, i.e.,
those for other devices. This means we will see a mismatch and resend
all indexes, but they will not. This is somewhat cleaner as it avoids
resending everything twice when two devices are upgraded, and in any
case, we have no reason to force a resend of incoming indexes here.

### Testing

It happens on my computer...
2024-04-08 11:14:27 +02:00
Jakob Borg
2e7c03420f
lib/db: Hold update lock while taking snapshot (#9496) 2024-04-05 21:32:43 +02:00
Jakob Borg
4dfb9d7c83 lib/api: Missing return after HTTP error 2024-03-21 08:57:43 -04:00
Emil Lundberg
2f15670094
lib/api: Extract session store (#9425)
This is an extract from PR #9175, which can be reviewed in isolation to
reduce the volume of changes to review all at once in #9175. There are
about to be several services and API handlers that read and set cookies
and session state, so this abstraction will prove helpful.

In particular a motivating cause for this is that with the current
architecture in PR #9175, in `api.go` the [`webauthnService` needs to
access the
session](https://github.com/syncthing/syncthing/pull/9175/files#diff-e2e14f22d818b8e635572ef0ee7718dee875c365e07225d760a6faae8be7772dR309-R310)
for authentication purposes but needs to be instantiated before the
`configMuxBuilder` for config purposes, because the WebAuthn additions
to config management need to perform WebAuthn registration ceremonies,
but currently the session management is embedded in the
`basicAuthAndSessionMiddleware` which is [instantiated much
later](https://github.com/syncthing/syncthing/pull/9175/files#diff-e2e14f22d818b8e635572ef0ee7718dee875c365e07225d760a6faae8be7772dL371-R380)
and only if authentication is enabled in `guiCfg`. This refactorization
extracts the session management out from `basicAuthAndSessionMiddleware`
so that `basicAuthAndSessionMiddleware` and `webauthnService` can both
use the same shared session management service to perform session
management logic.

### Testing

This is a refactorization intended to not change any externally
observable behaviour, so existing tests (e.g., `api_auth_test.go`)
should cover this where appropriate. I have manually verified that:

- Appending `+ "foo"` to the cookie name in `createSession` causes
`TestHtmlFormLogin/invalid_URL_returns_403_before_auth_and_404_after_auth`
and `TestHtmlFormLogin/UTF-8_auth_works` to fail
- Inverting the return value of `hasValidSession` cases a whole bunch of
tests in `TestHTTPLogin` and `TestHtmlFormLogin` to fail
- (Fixed) Changing the cookie to `MaxAge: 1000` in `destroySession` does
NOT cause any tests to fail!
- Added tests `TestHtmlFormLogin/Logout_removes_the_session_cookie`,
`TestHTTPLogin/*/Logout_removes_the_session_cookie`,
`TestHtmlFormLogin/Session_cookie_is_invalid_after_logout` and
`TestHTTPLogin/200_path#01/Session_cookie_is_invalid_after_logout` to
cover this.
- Manually verified that these tests pass both before and after the
changes in this PR, and that changing the cookie to `MaxAge: 1000` or
not calling `m.tokens.Delete(cookie.Value)` in `destroySession` makes
the respective pair of tests fail.
2024-03-21 08:09:47 -04:00
Simon Frei
8c8167a4ab
lib/model: Don't bump seq on error in index handler (#9459) 2024-03-11 07:30:21 +01:00
Simon Frei
73cc5553b6
lib/model: Prevent infinite index sending loop (fixes #9407) (#9458)
Explanation of what/why in a code comment.

Fixes https://github.com/syncthing/syncthing/issues/9407
2024-03-10 22:28:40 +01:00
Simon Frei
2ab2488274
lib/scanner: Fix ticker leak in scanner (fixes #9417) (#9451)
Move the ticker closer to where it's used and defer stop it to avoid
missing a branch.

Fixes regression introduced in
2f3eacdb6c

Fixes https://github.com/syncthing/syncthing/issues/9417
2024-03-05 19:04:26 +01:00
Luke Hamburg
bbd2a7fbc5
lib/model: Ignore difference in extended attributes & ownership when deleting (fixes #9371) (#9430)
Adds a bool flag to `scanIfItemChanged()` to indicate when the scan was initiated from a delete function, and if so, tell `IsEquivalentOptional()` to ignore Xattrs and Ownership regardless of the global setting.

I tested this with my sledgehammer and it seems to pass.
2024-03-02 14:55:18 +00:00
Jakob Borg
07a9fa2dbd
all: Use own automaxprocs package that doesn't log (ref #9436) (#9437)
### Purpose

🤫
2024-02-27 13:05:19 +01:00
Jakob Borg
ad81ac8da7
lib/api: Deflake TestAPIServiceRequests (#9413)
Somewhere along the way, the non-parallel test became parallel, and at
that point, timeouts occurred. Parallel is better, so increase the
timeout on the offending call a bit...
2024-02-11 09:20:29 +01:00
Jakob Borg
7ebeaefe77
lib/model: Deflake new IndexHandlerTest (#9412)
The new test has a flakiness factor on slow platforms, where the close
on the sending connection races with the last index message, potentially
messing up the count. This adds a wait to ensure that all sent messages
are received, or the test will eventually fail with a timeout.
2024-02-11 09:03:12 +01:00
Jakob Borg
e1dd36561d
all: Use some Go 1.21 features (#9409) 2024-02-10 21:02:42 +01:00
Jakob Borg
96c30f8387
lib/model, lib/protocol: Remove FileInfoBatch reuse behavior (#9399) 2024-02-10 19:16:27 +01:00
Jakob Borg
fc8b353011
build: Use Go 1.22, minimum is Go 1.21 (#9408)
Also updated dependencies, and an adjustment to build tags for how those
are handled and how irrelevant go1.15 is nowadays...
2024-02-09 16:35:29 +01:00
Jakob Borg
416b9e8924
lib/logger: Reduce API surface (#9404)
There is no need to expose the IsTraced() thing; it's just used in
initialisation, and thereafter ShouldDebug() is the corresponding
correct call.
2024-02-09 11:17:44 +01:00
gudvinr
9f6d732587
lib/logger: Split STTRACE into list of strings (#9402)
Currently `IsTraced("xyz")` will return true for
any inclusion of "xyz" in string.

This change splits `STTRACE` using `','`, `' '` and `';'`
as delimiters. That makes facilities separation
more clear.
2024-02-06 14:07:59 +01:00
Jakob Borg
eb617865d2 lib/model: Typo in method name (fixes #9389) 2024-02-01 15:13:33 +01:00
Jakob Borg
a49e318d25 lib/model: Typo in debug print (fixes #9386) 2024-02-01 15:11:09 +01:00
Jakob Borg
f16817632f
lib/api: Improve folder summary event, verbose service (#9370)
This makes a couple of small improvements to the folder summary
mechanism:

- The folder summary includes the local and remote sequence numbers in
clear text, rather than some odd sum that I'm not sure what it was
intended to represent.
- The folder summary event is generated when appropriate, regardless of
whether there is an event listener. We did this before because
generating it was expensive, and we wanted to avoid doing it
unnecessarily. Nowadays, however, it's mostly just reading out
pre-calculated metadata, and anyway, it's nice if it shows up reliably
when running with -verbose.

The point of all this is to make it easier to use these events to judge
when devices are, in fact, in sync. As-is, if I'm looking at two
devices, it's very difficult to reliably determine if they are in sync
or not. The reason is that while we can ask device A if it thinks it's
in sync, we can't see if the answer is "yes" because it has processed
all changes from B, or if it just doesn't know about the changes from B
yet. With proper sequence numbers in the event we can compare the two
and determine the truth. This makes testing a lot easier.
2024-01-31 08:24:39 +01:00
Jakob Borg
bda4016109
lib/protocol: Refactor interface (#9375)
This is a refactor of the protocol/model interface to take the actual
message as the parameter, instead of the broken-out fields:

```diff
type Model interface {
        // An index was received from the peer device
-       Index(conn Connection, folder string, files []FileInfo) error
+       Index(conn Connection, idx *Index) error
        // An index update was received from the peer device
-       IndexUpdate(conn Connection, folder string, files []FileInfo) error
+       IndexUpdate(conn Connection, idxUp *IndexUpdate) error
        // A request was made by the peer device
-       Request(conn Connection, folder, name string, blockNo, size int32, offset int64, hash []byte, weakHash uint32, fromTemporary bool) (RequestResponse, error)
+       Request(conn Connection, req *Request) (RequestResponse, error)
        // A cluster configuration message was received
-       ClusterConfig(conn Connection, config ClusterConfig) error
+       ClusterConfig(conn Connection, config *ClusterConfig) error
        // The peer device closed the connection or an error occurred
        Closed(conn Connection, err error)
        // The peer device sent progress updates for the files it is currently downloading
-       DownloadProgress(conn Connection, folder string, updates []FileDownloadProgressUpdate) error
+       DownloadProgress(conn Connection, p *DownloadProgress) error
 }
```

(and changing the `ClusterConfig` to `*ClusterConfig` for symmetry;
we'll be forced to use all pointers everywhere at some point anyway...)

The reason for this is that I have another thing cooking which is a
small troubleshooting change to check index consistency during transfer.
This required adding a field or two to the index/indexupdate messages,
and plumbing the extra parameters in umpteen changes is almost as big a
diff as this is. I figured let's do it once and avoid having to do that
in the future again...

The rest of the diff falls out of the change above, much of it being in
test code where we run these methods manually...
2024-01-31 08:18:27 +01:00
kylosus
302b352d78
lib/fs: Add invalid UTF-8 guards to watcher (fixes #9369) (#9372)
Add invalid UTF-8 guards to fix #9369. Probably not a permanent fix, but
putting it up here in case someone else encounters the same panic.
2024-01-28 19:50:26 +01:00
Jakob Borg
45beb28fa5 lib/api: Remove remnants of CSRF tokens file mentions (ref #9284) 2024-01-23 12:07:58 +01:00
Jakob Borg
3297624037
lib/ignore: Optimise ignoring directories for filesystem watcher (fixes #9339) (#9340)
This improves the ignore handling so that directories can be fully
ignored (skipped in the watcher) in more cases. Specifically, where the
previous rule was that any complex `!`-pattern would disable skipping
directories, the new rule is that only matches on patterns *after* such
a `!`-pattern disable skipping. That is, the following now does the
intuitive thing:

```
/foo
/bar
!whatever
*
```

- `/foo/**` and `/bar/**` are completely skipped, since there is no
chance anything underneath them could ever be not-ignored
- `!whatever` toggles the "can't skip directories any more" flag
- Anything that matches `*` can't skip directories, because it's
possible we can have `whatever` match something deeper.

To enable this, some refactoring was necessary:

- The "can skip dirs" flag is now a property of the match result, not of
the pattern set as a whole.
- That meant returning a boolean is not good enough, we need to actually
return the entire `Result` (or, like, two booleans but that seemed
uglier and more annoying to use)
- `ShouldIgnore(string) boolean` went away with
`Match(string).IsIgnored()` being the obvious replacement (API
simplification!)
- The watcher then needed to import the `ignore` package (for the
`Result` type), but `fs` imports the watcher and `ignore` imports `fs`.
That's a cycle, so I broke out `Result` into a package of its own so
that it can be safely imported everywhere in things like `type Matcher
interface { Match(string) result.Result }`. There's a fair amount of
stuttering in `result.Result` and maybe we should go with something like
`ignoreresult.R` or so, leaving this open for discussion.

Tests refactored to suit, I think this change is in fact quite well
covered by the existing ones...

Also some noise because a few of the changed files were quite old and
got the `gofumpt` treatment by my editor. Sorry not sorry.

---------

Co-authored-by: Simon Frei <freisim93@gmail.com>
2024-01-15 10:13:22 +00:00
Jakob Borg
e041877488
lib/ignore: Refactor out result type (#9343) 2024-01-13 18:58:23 +01:00
nf
8b321387c0
lib/versioner: Expand tildes in version directory (fixes #9241) (#9327)
### Purpose

Fix #9241 by expanding tildes in version paths.

When creating the versioner file system, first try to expand any leading
tildes to the user's home directory before handling relative paths. This
makes a version path `"~/p"` expand to `"$HOME/p"` instead of
`"/folder/~/p"`.

### Testing

Added a test to lib/versioner that exercises this code path. Also
manually tested with local syncthing instances.
2024-01-12 10:46:18 +01:00
Julian Lehrhuber
8edd67a569
lib/scanner: Prevent sync-conflict for receive-only local modifications (#9323)
### Purpose

This PR changes behaviour of syncthing related to `receive-only`
folders, which I believe to be a bug since I wouldn't expect the current
behaviour. With the current syncthing codebase, a file of a
`receive-only` folder that is only modified locally can cause the
creation of a `.sync-conflict` file.

### Testing

Consider this szenario: Setup two paired clients that sync a folder with
a given file (e.g. `Test.txt`). One of the clients configures the folder
to be `receive-only`. Now, change the contents of the file for the
receive-only client **_twice_**.

With the current syncthing codebase, this leads to the creation of a
`.sync-conflict` file that contains the modified contents, while the
regular `Test.txt` file is reset to the cluster's provided contents.
This is due to a `protocol.FileInfo#ShouldConflict` check, that is
succeeding on the locally modified file.

This PR changes this behaviour to not reset the file and not cause the
creation of a `.sync-conflict`. Instead, the second content update is
treated the same as the first content update.

This PR also contains a test that fails on the current codebase and
succeeds with the changes introduced in this PR.

### Screenshots

This is not a GUI change

### Documentation

This is not a user visible change.

## Authorship

Your name and email will be added automatically to the AUTHORS file
based on the commit metadata.

#### Thanks to all the syncthing folks for this awesome piece of
software!
2024-01-08 10:29:20 +01:00
Jakob Borg
aa901790b9
lib/api: Save session & CSRF tokens to database, add option to stay logged in (fixes #9151) (#9284)
This adds a "token manager" which handles storing and checking expired
tokens, used for both sessions and CSRF tokens. It removes the old,
corresponding functionality for CSRFs which saved things in a file. The
result is less crap in the state directory, and active login sessions
now survive a Syncthing restart (this really annoyed me).

It also adds a boolean on login to create a longer-lived session cookie,
which is now possible and useful. Thus we can remain logged in over
browser restarts, which was also annoying... :)

<img width="1001" alt="Screenshot 2023-12-12 at 09 56 34"
src="https://github.com/syncthing/syncthing/assets/125426/55cb20c8-78fc-453e-825d-655b94c8623b">

Best viewed with whitespace-insensitive diff, as a bunch of the auth
functions became methods instead of closures which changed indentation.
2024-01-04 10:07:12 +00:00
Peter Badida
fc1c7a3c49
lib/build: Allow semver build in version regex (fixes #9267) (#9316) 2024-01-02 20:43:22 +01:00
Peter Badida
2abfefc18c
gui: Keep short deviceID length consistent + xrefs (fixes #9313) (#9314)
Making short deviceID length consistent and referencing to protocol file
for future-proof edits. Closes #9313.
2024-01-02 17:31:57 +01:00
Simon Frei
2f3eacdb6c
gui, lib/scanner: Improve scan progress indication (ref #8331) (#9308) 2023-12-31 23:01:16 +01:00
Sven Bachmann
1ce2af1238
lib/protocol: handle empty names in unixOwnershipEqual (fixes #9039) (#9306)
If syncOwnership is enabled and the remote uses for example a dockerized
Syncthing it can't fetch the ownername and groupname of the local
instance. Without this patch this led to an endless cycle of detected
changes on the remote and failing re-sync attempts.

This patch skips comparing the ownername and groupname if they zare empty
on one side.

See https://github.com/syncthing/syncthing/issues/9039 for details.

### Testing

Proposed by @calmh in
https://github.com/syncthing/syncthing/issues/9039#issuecomment-1870584783
and tested locally in my setup,

Setup PC 1:
- Syncthing is run in Docker as user `root` and has none of the users
configured that synchronize their files

Setup PC 2:
  - this PC has all users locally setup
- Syncthing runs as `systemd` service as user `syncthing` and has
multiple capabilities set to set the correct owner and permissions

Setup PC 3:
  - same as PC 2

Handling:
- `PC 1` is send & receive and uses just the `UID` and `GID` identifiers
to store the files
- `PC 2` and `PC 3` synchronize their files over `PC 1` but not directly
to each other

Outcome:
- `PC 2` and `PC 3` should send and receive their files with the correct
ownership and groups from `PC 1`
2023-12-29 09:16:33 +01:00
greatroar
cdefa535ed
lib/connections: Skip allocation in check for missing port (#9297)
Micro-optimization. Already has unit tests.
2023-12-20 11:59:11 +01:00
gudvinr
91084b83b4
lib/upgrade: Extract signing key to embedded file (fixes #9247) (#9296)
### Purpose

Instead of hardcoding `SigningKey` as text use `go:embed`. Fixes #9247.

### Testing

* Building syncthing
* Trying to upgrade (signature verification)
2023-12-18 19:47:57 +00:00
Eric P
e8d3529fed
lib/model: Only handle relevant folder summaries (kqueue) (fixes #9183) (#9288)
On kqueue-systems, folders listen for folder summaries to (be able to)
warn for potential high resource usage. However, it listened for any
folder summary and not for the summary which matches the folder it's
about. This could cause that an unwatched folder causes a folder summary
containing more files than the threshold (10k), and the listening folder
(with the watcher enabled) triggers the warning.

This makes sure that only the folder summaries which are relevant to the
specific folder are being handled.

### Testing

- Fire up some kqueue-system (freebsd, I used).
- add folder A, disable the watcher, add 10001 files
- add folder B with the watcher enabled, no files are needed here

Before the change:
- add an item to folder A, trigger a rescan to speed up the process
- wait some seconds...warning triggered by folder B's
summarySubscription

After the change:
- Only a warning is triggered if the received folder summary matches the
folder which listens for the summaries
2023-12-13 12:34:24 +01:00
Jakob Borg
935a28c961
lib/model: Use a single lock (phase two: cleanup) (#9276)
Cleanup after #9275.

This renames `fmut` -> `mut`, removes the deadlock detector and
associated plumbing, renames some things from `...PRLocked` to
`...RLocked` and similar, and updates comments.

Apart from the removal of the deadlock detection machinery, no
functional code changes... i.e. almost 100% diff noise, have fun
reviewing.
2023-12-11 22:06:45 +01:00
Jakob Borg
6f1023665c
lib/model: Use a single lock (#9275)
I'm tired of the fmut/pmut shenanigans. This consolidates both under one
lock; I'm not convinced there are any significant performance
differences with this approach since we're literally just protecting map
juggling...

- The locking goes away when we were already under an appropriate fmut
lock.
- Where we had fmut.RLock()+pmut.Lock() it gets upgraded to an
fmut.Lock().
- Otherwise s/pmut/fmut/.

In order to avoid diff noise for an important change I did not do the
following cleanups, which will be filed in a PR after this one, if
accepted:

- Renaming fmut to just mut
- Renaming methods that refer to being "PRLocked" and stuff like that
- Removing the no longer relevant deadlock detector
- Comments referring to pmut and locking sequences...
2023-12-11 21:26:23 +01:00
Jakob Borg
a2cbc62521 lib/nat: Fix test build failure (ref #9010) 2023-12-11 10:13:14 +01:00
Jakob Borg
30fe2cf514 lib/model: Add pmut locking for DeviceStatistics (fixes #9274)
Looking at deviceConnIDs requires this. Added in #9256.
2023-12-11 08:04:47 +01:00
Jakob Borg
d0e407f3c3 lib/model: Remove spurious "replacing service" failure event (ref #9271)
This is no longer a notable condition, as we do this pretty much all the
time.
2023-12-11 07:43:40 +01:00
Maximilian
16db6fcf3d
lib/nat, lib/upnp: IPv6 UPnP support (#9010)
This pull request allows syncthing to request an IPv6
[pinhole](https://en.wikipedia.org/wiki/Firewall_pinhole), addressing
issue #7406. This helps users who prefer to use IPv6 for hosting their
services or are forced to do so because of
[CGNAT](https://en.wikipedia.org/wiki/Carrier-grade_NAT). Otherwise,
such users would have to configure their firewall manually to allow
syncthing traffic to pass through while IPv4 users can use UPnP to take
care of network configuration already.

### Testing

I have tested this in a virtual machine setup with miniupnpd running on
the virtualized router. It successfully added an IPv6 pinhole when used
with IPv6 only, an IPv4 port mapping when used with IPv4 only and both
when dual-stack (IPv4 and IPv6) is used.

Automated tests could be added for SOAP responses from the router but
automatically testing this with a real network is likely infeasible.

### Documentation

https://docs.syncthing.net/users/firewall.html could be updated to
mention the fact that UPnP now works with IPv6, although this change is
more "behind the scenes".

---------

Co-authored-by: Simon Frei <freisim93@gmail.com>
Co-authored-by: bt90 <btom1990@googlemail.com>
Co-authored-by: André Colomb <github.com@andre.colomb.de>
2023-12-11 07:36:18 +01:00
Simon Frei
a28de73031
lib/model: Remove runner during folder cleanup (fixes #9269) (#9271)
Before introducing the service map and using it for folder runners, the
entries in folderCfgs and folderRunners for the same key/folder were
removed under a single lock. Stopping the folder happens separately
before that with just the read lock. Now with the service map stopping
the folder and removing it from the map is a single operation. And that
still happens with just a read-lock. However even with a full lock it’s
still problematic: After the folder stopped, the runner isn’t present
anymore while the folder-config still is and sais the folder isn't
paused.

The index handler in turn looks at the folder config that is not paused,
thus assumes the runner has to be present -> nil deref on the runner.

A better solution might be to push most of these fmut maps into the
folder - they anyway are needed in there. Then there's just a single
map/source of info that's necessarily consistent. That's quite a bit of
work though, and probably/likely there will be corner cases there too.
2023-12-08 07:13:09 +01:00
Jakob Borg
c1ec9a8826
lib/fs: Reduce memory usage in xattrs handling (#9251)
This reduces allocations, in number and in size, while getting extended
attributes. This is mostly noticable when there is a large number of new
files to scan and we're running with the default scanProgressInterval --
then a queue of files is built in-memory, and this queue includes
extended attributes as part of file metadata. (Arguable it shouldn't,
but that's a more difficult and involved change.)

With 1M files to scan, each with one extended attribute, current peak
memory usage looks like this:

	Showing nodes accounting for 1425.30MB, 98.19% of 1451.64MB total
	Dropped 1435 nodes (cum <= 7.26MB)
	Showing top 10 nodes out of 54
	      flat  flat%   sum%        cum   cum%
976.56MB 67.27% 67.27% 976.56MB 67.27%
github.com/syncthing/syncthing/lib/fs.getXattr
305.44MB 21.04% 88.31% 305.44MB 21.04%
github.com/syncthing/syncthing/lib/scanner.(*walker).walk.func1
45.78MB 3.15% 91.47% 1045.23MB 72.00%
github.com/syncthing/syncthing/lib/fs.(*BasicFilesystem).GetXattr
22.89MB 1.58% 93.04% 22.89MB 1.58%
github.com/syncthing/syncthing/lib/fs.listXattr
22.89MB 1.58% 94.62% 22.89MB 1.58%
github.com/syncthing/syncthing/lib/protocol.(*PlatformData).SetXattrs
16MB 1.10% 95.72% 16.01MB 1.10%
github.com/syndtr/goleveldb/leveldb/memdb.New

After the change, it's this:

	Showing nodes accounting for 502.32MB, 95.70% of 524.88MB total
	Dropped 1400 nodes (cum <= 2.62MB)
	Showing top 10 nodes out of 91
	      flat  flat%   sum%        cum   cum%
305.43MB 58.19% 58.19% 305.43MB 58.19%
github.com/syncthing/syncthing/lib/scanner.(*walker).walk.func1
45.79MB 8.72% 66.91% 68.68MB 13.09%
github.com/syncthing/syncthing/lib/fs.(*BasicFilesystem).GetXattr
32MB 6.10% 73.01% 32.01MB 6.10%
github.com/syndtr/goleveldb/leveldb/memdb.New
22.89MB 4.36% 77.37% 22.89MB 4.36%
github.com/syncthing/syncthing/lib/fs.listXattr
22.89MB 4.36% 81.73% 22.89MB 4.36%
github.com/syncthing/syncthing/lib/protocol.(*PlatformData).SetXattrs
15.35MB 2.92% 84.66% 15.36MB 2.93%
github.com/syndtr/goleveldb/leveldb/util.(*BufferPool).Get
	   15.28MB  2.91% 87.57%    15.28MB  2.91%  strings.(*Builder).grow

(The usage for xattrs is reduced from 976 MB to 68 MB)
2023-12-04 12:48:17 +01:00
Jakob Borg
1625b44892
lib/model: Improve LastSeen handling (#9256)
LastSeen for a device was only updated when they connected. This now
updates it when they disconnect, so that we remember the last time we
actually saw them. When asking the API for current stats, currently
connected devices get a last seen value of the current time.
2023-12-04 09:24:10 +01:00
Jakob Borg
d51760f410
lib/scanner: Record inode change time for directories and symlinks (#9250) 2023-12-04 07:48:24 +01:00
Jakob Borg
7b1932d64e
lib/api: Improve ignore loading error handling (fixes #9253) (#9254) 2023-12-04 07:11:35 +01:00
Jakob Borg
4cba99fcd4 lib/fs: Better equality comparison in mtimefs 2023-12-03 16:01:46 +01:00
Simon Frei
958ff67ccc
lib/model: Acquire fmut lock in ensureIndexHandler (fixes #9234) (#9235)
Towards the end of the function f.folderCfgs and f.folderRunners are
read without the lock.
2023-11-20 08:12:25 +01:00
Jakob Borg
439c6c5b7c
lib/api: Add cache busting for basic auth (ref #9208) (#9215)
This adds our short device ID to the basic auth realm. This has at least
two consequences:

- It is different from what's presented by another device on the same
address (e.g., if I use SSH forwards to different dives on the same
local address), preventing credentials for one from being sent to
another.

- It is different from what we did previously, meaning we avoid cached
credentials from old versions interfering with the new login flow.

I don't *think* there should be things that depend on our precise realm
string, so this shouldn't break any existing setups...

Sneakily this also changes the session cookie and CSRF name, because I
think `id.Short().String()` is nicer than `id.String()[:5]` and the
short ID is two characters longer. That's also not a problem...
2023-11-14 11:57:39 +01:00
Jakob Borg
8f1b0df74b
lib/api: Improve cookie handling (fixes #9208) (#9214) 2023-11-13 20:37:29 +01:00
Jakob Borg
bae6d5f375
lib/location: Fix regression of timestamp handling (ref #9180) (#9185) 2023-10-26 07:41:02 +02:00
Jakob Borg
b5082f6af8
lib/locations: Change default config/data location to new XDG recommendation (fixes #9178, fixes #9179) (#9180)
This makes the new default $XDG_STATE_HOME/syncthing or
~/.local/state/syncthing, while still looking in legacy locations first
for existing installs.

Note that this does not *move* existing installs, and nor should we.
Existing paths will continue to be used as-is, but the user can move the
dir into the new place if they want to use it (as they could prior to
this change as well, for that matter).

### Documentation

Needs update to the config docs about our default locations.
2023-10-25 11:16:24 +02:00
tomasz1986
16ae1fbe5e
lib/fs: Ignore inode change time on Android (#9177)
lib/fs: Fix conflicts on Android due to fluctuating inode change time

[1] added inode change time to file info in order to support syncing
extended attributes. However, in the case of Android, this inode change
time fluctuates, leading to unexpected conflicts even when the user has
not even touched the files on the Android device itself. Thus, in order
to prevent those conflicts from happening, do not write inode change
time on Android.

[1] 6cac308bcd

Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
2023-10-21 08:24:29 +02:00
Emil Lundberg
14569f12d3
Hide log out button when auth is not enabled (#9158)
This was an oversight in #8757: the new "Log out" button is always shown
in the "Actions" menu, even when authentication is not enabled.
2023-10-15 14:10:41 +02:00
Eric P
9553365d31
lib/fs: Properly handle Windows deduplicated files (fixes #9120) (#9168)
### Purpose

Deduplicated files are apparently considered 'irregular' under the hood,
this causes them to simply be ignored by Syncthing. This change is more
of a workaround than a proper fix, as the fix should probably happen in
the underlying libraries? - which may take some time. In the meanwhile,
this change should make deduplicated files be treated as regular files
and be indexed and synced as they should.

### Testing

Create some volume where deduplication is turned on (see the relevant
issue for details, including a proper description of how to reproduce
it). Prior to this change, the deduplicated files were simply ignored
(even by the indexer). After this change, the deduplicated files are
being index and synced properly.
2023-10-11 14:40:55 +02:00
Emil Lundberg
ea1ea366d2 lib/api: Check basic auth (and set session cookie) before noauth exceptions (#9159)
This is motivated by the Android app:
https://github.com/syncthing/syncthing-android/pull/1982#issuecomment-1752042554

The planned fix in response to basic auth behaviour changing in #8757
was to add the `Authorization` header when opening the WebView, but it
turns out the function used only applies the header to the initial page
load, not any subsequent script loads or AJAX calls. The
`basicAuthAndSessionMiddleware` checks for no-auth exceptions before
checking the `Authorization` header, so the header has no effect on the
initial page load since the `/` path is a no-auth exception. Thus the
Android app fails to log in when opening the WebView.

This changes the order of checks in `basicAuthAndSessionMiddleware` so
that the `Authorization` header is always checked if present, and a
session cookie is set if it is valid. Only after that does the
middleware fall back to checking for no-auth exceptions.

`api_test.go` has been expanded with additional checks:
- Check that a session cookie is set whenever correct basic auth is
provided.
- Check that a session cookie is not set when basic auth is incorrect.
- Check that a session cookie is not set when authenticating with an API
token (either via `X-Api-Key` or `Authorization: Bearer`).

And an additional test case:
- Check that requests to `/` always succeed, but receive a session
cookie when correct basic auth is provided.

I have manually verified that
- The new assertions fail if the `createSession` call is removed in
`basicAuthAndSessionMiddleware`.
- The new test cases in e6e4df4d7034302b729ada6d91cff6e2b29678da fail
before the change in 0e47d37e738d4c15736c496e01cd949afb372e71 is
applied.
2023-10-10 07:48:55 +02:00
Jakob Borg
3d0da5ac60
lib/api: Better handle %s templates in LDAP strings (fixes #9072) (#9155)
Also add some escaping for good measure.
2023-10-07 02:29:53 +00:00
Jakob Borg
a64ae36bcc
lib/model: Verify versioning on configuration reload (fixes #9106) (#9154) 2023-10-07 04:09:51 +02:00
Emil Lundberg
8294870ffc
Add HTML login form (fixes #4137) (#8757) 2023-10-06 13:00:58 +02:00
Jakob Borg
296db314f5
lib/config: Improve parsing of gui-address overrides (#9144)
improve parsing of gui-address overrides

make checks for whether the gui-address is overridden consistent by
checking whether the environment variable is set and not an empty
string. the `Network()` function however checked for the inclusion of
a slash instead of the presence of any characters. If the config file's
gui address was set to a unix socket and the gui override to a tcp
address, then the function would have wrongly returned "unix".

the `URL()` function always returned the config file's gui address if a
unix socket was configured, even if an override was specified.

the `URL()` function wrongly formatted unix addresses. the http(s)
protocol was used as the sheme and the path was percent escaped. because
of the previous bug, this could only be triggered if the config file's
gui address was tcp and an unix socket override was given.

simplify the `useTLS()` function's codepath for overrides.

Co-authored-by: digital <didev@dinid.net>
2023-10-02 08:40:03 +02:00
Jakob Borg
b91d7711aa
Update dependencies (#9129)
And some QUIC API changes, of course.
2023-09-25 21:45:57 +02:00
Jakob Borg
6ed9c0c34c
lib/config: Accept pre-hashed password (fixes #9123) (#9124) 2023-09-24 19:23:49 +02:00
bt90
cf46bf0297
lib/connections: Fix transport type detection for QUIC (fixes #8274) (#9114)
Check remote address
2023-09-20 11:23:48 +02:00
Jakob Borg
051cbdc713
lib/fs, lib/model: Be careful about potentially negative durations (fixes #9112) (#9113)
I don't really understand under what circumstances, but sometimes these
calls panic with a "panic: counter cannot decrease in value" because the
value passed to Add() was negative.
2023-09-20 09:04:47 +02:00
Jakob Borg
f47de83914
lib/protocol: Ensure starting & closing a connection are exclusive (fixes #9102) (#9103)
In principle a connection can close while it's in progress with
starting, and then it's undefined if we wait for goroutines to exit etc.
With this change, we will wait for start to complete before starting to
stop everything.
2023-09-12 14:48:15 +02:00
bt90
e860d3b974
lib/connections: Make assumptions about isLAN when interface address listing fails (#9093) 2023-09-12 12:34:30 +00:00
bt90
ed66fba42b
lib/beacon, lib/discover: Send IPv4 limited broadcast when address listing fails (fixes #1628) (#9087) 2023-09-12 14:28:17 +02:00
Jakob Borg
4812600098
lib/versioner: Don't complain when folder is stopping (#9097) 2023-09-11 23:10:18 +02:00
Jakob Borg
7c0223bd06 lib/build: Next version is the Gold Grasshopper 2023-09-06 13:13:39 +02:00
Jakob Borg
c6334e61aa
all: Support multiple device connections (fixes #141) (#8918)
This adds the ability to have multiple concurrent connections to a single device. This is primarily useful when the network has multiple physical links for aggregated bandwidth. A single connection will never see a higher rate than a single link can give, but multiple connections are load-balanced over multiple links.

It is also incidentally useful for older multi-core CPUs, where bandwidth could be limited by the TLS performance of a single CPU core -- using multiple connections achieves concurrency in the required crypto calculations...

Co-authored-by: Simon Frei <freisim93@gmail.com>
Co-authored-by: tomasz1986 <twilczynski@naver.com>
Co-authored-by: bt90 <btom1990@googlemail.com>
2023-09-06 12:52:01 +02:00
Maximilian
c42c0e7ceb
lib/connections: Fix WANAddresses returning only unspecified IPs (ref #9010) (#9073)
Avoids taking the address of the same variable twice.
2023-09-03 15:03:27 +00:00
Jakob Borg
5118538179
lib/model: Refactor folderRunners to use a serviceMap (#9071)
Instead of separately tracking the token.

Also changes serviceMap to have a channel version of RemoveAndWait, so
that it's possible to do the removal under a lock but wait outside of
the lock. And changed where we do that in connection close, reversing
the change that happened when I added the serviceMap in 40b3b9ad1.
2023-09-02 16:42:46 +02:00
Jakob Borg
3130af3773
lib/upgrade: Enable HTTP/2 for upgrade checks (#9060) 2023-08-30 21:58:34 +02:00
Jakob Borg
abd89f15f7
lib/discover: Enable HTTP/2 for global discovery requests (#9059)
By creating the http.Transport and tls.Configuration ourselves we
override some default behavior and end up with a client that speaks only
HTTP/1.1.

This adds a call to http.ConfigureTransport to do the relevant magic to
enable HTTP/2.

Also tweaks the keepalive settings to be a little kinder to the
server(s).
2023-08-30 21:58:05 +02:00
bt90
467522d04d
lib/connections: Allow IPv6 ULA in discovery announcements (fixes #7456) (#9048)
The allowed IPv4 ranges are the same as before. But we now also accept IPv6 addresses in the ULA range FC00::/7. These addresses don't require an interface identifier and are roughly equivalent to the IPv4 private ranges.

Typical usecases:

VPN interface IPs: Wireguard, OpenVPN, Tailscale, ...
fixed IPv6 LAN addressing while the provider assigns a dynamic prefix. e.g used by pihole
https://cs.opensource.google/go/go/+/refs/tags/go1.21.0:src/net/ip.go;l=146
2023-08-23 12:28:48 +02:00
bt90
3147285c60
lib/beacon: Check FlagRunning (#9051) 2023-08-22 11:27:43 +02:00
Jakob Borg
acd767b30b
all: Remove lib/util package (#9049)
Grab-bag packages are nasty, this cleans it up a little by splitting it
into topical packages sempahore, netutil, stringutil, structutil.
2023-08-21 19:44:33 +02:00
Jakob Borg
40b3b9ad15
lib/model: Clean up index handler life cycle (fixes #9021) (#9038)
Co-authored-by: Simon Frei <freisim93@gmail.com>
2023-08-21 18:39:13 +02:00
bt90
c2c6133aa5
lib/osutil, lib/upnp: Check FlagRunning (fixes #8767) (#9047) 2023-08-21 14:49:28 +00:00
Jakob Borg
cbf0e31f69
all: Use Go 1.21, new QUIC API (#9040) 2023-08-21 15:25:52 +02:00
Jakob Borg
462389934b cmd/stupgrades: Serve friendlier URLs for upgrade assets (fixes #9033) 2023-08-09 21:01:15 +02:00
André Colomb
a8cacdca94
lib/versioner: Minor fixes in comments and error message (#9031)
* lib/versioner: Factor out DefaultPath constant.

Replace several instances where .stversions is named literally to all
use the same definition in the versioner package.  Exceptions are the
packages where a cyclic dependency on versioner is impossible, or some
tests which combine the versions base path with other components.

* lib/versioner: Fix comment about trash can in simple versioner.

* lib/versioner: Fix wrong versioning type string in error message.

The error message shows the folder type instead of the versioning
type, although the correct field is used in the comparison.
2023-08-09 07:10:06 +00:00
Jakob Borg
8b87cd5229
lib/model: Reinstate setting folder idle state (#9029) 2023-08-08 07:24:02 +02:00
Jakob Borg
b9c08d3814
all: Add Prometheus-style metrics to expose some internal performance counters (fixes #5175) (#9003) 2023-08-04 19:57:30 +02:00
Keith Harrison
eed12f3ec5
lib/config: Allow sharing already encrypted folder with untrusted devices (fixes #8965) (#9012)
Safety check added in v1.23.6 introduced bug. Bug unshares folders with untrusted devices if folder does not have an encryption password set, regardless of whether the folder is shared with the untrusted device as encrypted or not. Prevents sharing with untrusted devices in some cases where sharing would be encrypted.

Patch preserves safety check but permits sharing folders with untrusted devices if they are shared as encrypted.

Signed-off-by: kewiha <keithh@protonmail.com>
2023-08-02 07:14:53 +00:00
Jakob Borg
9d21b91124
all: Refactor the protocol/model interface a bit (ref #8981) (#9007) 2023-07-29 10:24:44 +02:00
Chih-Hsuan Yen
b806026990
lib/connections: Fix building with -tags noquic (#9009) 2023-07-28 10:08:50 +00:00
Emil Lundberg
b08b99e284
lib/api: Fix data race in TestCSRFRequired (#9006) 2023-07-26 21:33:45 +00:00
Jakob Borg
855c6dc67b
lib/api: Allow Bearer authentication style with API key (#9002)
Currently, historically, we look for the `X-API-Key` header to
authenticate with an API key. There's nothing wrong with this, but in
some scenarios it's easier to produce an `Authorization` header with a
`Bearer $token` content, which is nowadays more common. This change adds
support for both, so that we will accept an API key either in our custom
header or as a bearer token.
2023-07-26 13:13:06 +02:00
tomasz1986
f42f041f53
lib/ur: Don't report uptime if start time is in the past (fixes #7698) (#8996)
Currently, because of devices with unset RTC clock, the 100% percentile
for Uptime on [1] is calculated since the Unix epoch which is useless as
far as usage statistics are concerned. Thus, if the Syncthing start time
is set to a past date, assume that the clock is wrong and do not even
try to report the uptime.

[1] https://data.syncthing.net

Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
Co-authored-by: Jakob Borg <jakob@kastelo.net>
2023-07-22 21:25:03 +00:00
Christian Kujau
6b6b2c6194
lib/model: use WARN for "Unexpected folder" messages (#8998) 2023-07-22 21:17:32 +00:00
tomasz1986
d70eb569f2
lib/osutil: Skip setLowPriority in Windows if already lower (fixes #6597) (#8993) 2023-07-21 04:38:15 +00:00
deepsource-autofix[bot]
21c074cc2c
all: replace empty slice literal with var (#8990)
refactor: replace empty slice literal with `var`

An empty slice can be represented by `nil` or an empty slice literal. They are
functionally equivalent — their `len` and `cap` are both zero — but the `nil`
slice is the preferred style. For more information about empty slices,
see [Declaring Empty Slices](https://github.com/golang/go/wiki/CodeReviewComments#declaring-empty-slices).

Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>
2023-07-18 14:44:37 +00:00
deepsource-autofix[bot]
f23c41221b
all: fix unused method receiver (#8988)
refactor: fix unused method receiver

Methods with unused receivers can be a symptom of unfinished refactoring or a bug. To keep 
the same method signature, omit the receiver name or '_' as it is unused.

Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>
2023-07-18 14:34:50 +00:00
deepsource-autofix[bot]
24e230d455
all: unused parameter should be replaced by underscore (#8989)
refactor: unused parameter should be replaced by underscore

Unused parameters in functions or methods should be replaced with `_`
(underscore) or removed.

Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>
2023-07-18 14:33:13 +00:00
Jakob Borg
df2ac7aaeb
gui, lib/api: Add possibility to feed through extra version information (#8980)
This adds an environment variable STVERSIONEXTRA that, when set, gets
added to the version information in the API and GUI.

The purpose of all this is to be able to communicate something about the
bundling or packaging, through the log & GUI and the end user, to the
potential person supporting it -- i.e., us. :) A wrapper can set this
variable to indicate that Syncthing is being run via `SyncTrayzor`,
`Syncthing-macOS`, etc., and thus indicate to the end user that the GUI
they are looking at is perhaps not the only source of truth and
management for this instance.
2023-07-16 17:43:10 +02:00
Jakob Borg
c44de2cd58
lib/fs: Clarify errors for Windows filenames (fixes #8968) (#8969)
With this change, error messages include the offending characters or
name parts. Examples:

    nul.txt: name is invalid, contains Windows reserved name: "nul"
    foo>bar.txt: name is invalid, contains Windows reserved character: ">"
    foo \bar.txt: name is invalid, must not end in space or period on Windows
2023-07-07 11:00:40 +00:00
guangwu
27aba3567b
all: Minor staticcheck fixes (#8939) 2023-06-19 06:50:53 +00:00
Jakob Borg
6b475bdb78
lib/config, gui: Disallow some options in combination with "untrusted" (fixes #8920) (#8921)
This prevents combining untrusted with introducer and auto-accept, and
also verifies that folders shared with untrusted devices have passwords
at config loading time.

Co-authored-by: Simon Frei <freisim93@gmail.com>
2023-06-14 09:24:31 +02:00