Commit Graph

869 Commits

Author SHA1 Message Date
Julian Lehrhuber
8edd67a569
lib/scanner: Prevent sync-conflict for receive-only local modifications (#9323)
### Purpose

This PR changes behaviour of syncthing related to `receive-only`
folders, which I believe to be a bug since I wouldn't expect the current
behaviour. With the current syncthing codebase, a file of a
`receive-only` folder that is only modified locally can cause the
creation of a `.sync-conflict` file.

### Testing

Consider this szenario: Setup two paired clients that sync a folder with
a given file (e.g. `Test.txt`). One of the clients configures the folder
to be `receive-only`. Now, change the contents of the file for the
receive-only client **_twice_**.

With the current syncthing codebase, this leads to the creation of a
`.sync-conflict` file that contains the modified contents, while the
regular `Test.txt` file is reset to the cluster's provided contents.
This is due to a `protocol.FileInfo#ShouldConflict` check, that is
succeeding on the locally modified file.

This PR changes this behaviour to not reset the file and not cause the
creation of a `.sync-conflict`. Instead, the second content update is
treated the same as the first content update.

This PR also contains a test that fails on the current codebase and
succeeds with the changes introduced in this PR.

### Screenshots

This is not a GUI change

### Documentation

This is not a user visible change.

## Authorship

Your name and email will be added automatically to the AUTHORS file
based on the commit metadata.

#### Thanks to all the syncthing folks for this awesome piece of
software!
2024-01-08 10:29:20 +01:00
Eric P
e8d3529fed
lib/model: Only handle relevant folder summaries (kqueue) (fixes #9183) (#9288)
On kqueue-systems, folders listen for folder summaries to (be able to)
warn for potential high resource usage. However, it listened for any
folder summary and not for the summary which matches the folder it's
about. This could cause that an unwatched folder causes a folder summary
containing more files than the threshold (10k), and the listening folder
(with the watcher enabled) triggers the warning.

This makes sure that only the folder summaries which are relevant to the
specific folder are being handled.

### Testing

- Fire up some kqueue-system (freebsd, I used).
- add folder A, disable the watcher, add 10001 files
- add folder B with the watcher enabled, no files are needed here

Before the change:
- add an item to folder A, trigger a rescan to speed up the process
- wait some seconds...warning triggered by folder B's
summarySubscription

After the change:
- Only a warning is triggered if the received folder summary matches the
folder which listens for the summaries
2023-12-13 12:34:24 +01:00
Jakob Borg
935a28c961
lib/model: Use a single lock (phase two: cleanup) (#9276)
Cleanup after #9275.

This renames `fmut` -> `mut`, removes the deadlock detector and
associated plumbing, renames some things from `...PRLocked` to
`...RLocked` and similar, and updates comments.

Apart from the removal of the deadlock detection machinery, no
functional code changes... i.e. almost 100% diff noise, have fun
reviewing.
2023-12-11 22:06:45 +01:00
Jakob Borg
6f1023665c
lib/model: Use a single lock (#9275)
I'm tired of the fmut/pmut shenanigans. This consolidates both under one
lock; I'm not convinced there are any significant performance
differences with this approach since we're literally just protecting map
juggling...

- The locking goes away when we were already under an appropriate fmut
lock.
- Where we had fmut.RLock()+pmut.Lock() it gets upgraded to an
fmut.Lock().
- Otherwise s/pmut/fmut/.

In order to avoid diff noise for an important change I did not do the
following cleanups, which will be filed in a PR after this one, if
accepted:

- Renaming fmut to just mut
- Renaming methods that refer to being "PRLocked" and stuff like that
- Removing the no longer relevant deadlock detector
- Comments referring to pmut and locking sequences...
2023-12-11 21:26:23 +01:00
Jakob Borg
30fe2cf514 lib/model: Add pmut locking for DeviceStatistics (fixes #9274)
Looking at deviceConnIDs requires this. Added in #9256.
2023-12-11 08:04:47 +01:00
Jakob Borg
d0e407f3c3 lib/model: Remove spurious "replacing service" failure event (ref #9271)
This is no longer a notable condition, as we do this pretty much all the
time.
2023-12-11 07:43:40 +01:00
Simon Frei
a28de73031
lib/model: Remove runner during folder cleanup (fixes #9269) (#9271)
Before introducing the service map and using it for folder runners, the
entries in folderCfgs and folderRunners for the same key/folder were
removed under a single lock. Stopping the folder happens separately
before that with just the read lock. Now with the service map stopping
the folder and removing it from the map is a single operation. And that
still happens with just a read-lock. However even with a full lock it’s
still problematic: After the folder stopped, the runner isn’t present
anymore while the folder-config still is and sais the folder isn't
paused.

The index handler in turn looks at the folder config that is not paused,
thus assumes the runner has to be present -> nil deref on the runner.

A better solution might be to push most of these fmut maps into the
folder - they anyway are needed in there. Then there's just a single
map/source of info that's necessarily consistent. That's quite a bit of
work though, and probably/likely there will be corner cases there too.
2023-12-08 07:13:09 +01:00
Jakob Borg
1625b44892
lib/model: Improve LastSeen handling (#9256)
LastSeen for a device was only updated when they connected. This now
updates it when they disconnect, so that we remember the last time we
actually saw them. When asking the API for current stats, currently
connected devices get a last seen value of the current time.
2023-12-04 09:24:10 +01:00
Simon Frei
958ff67ccc
lib/model: Acquire fmut lock in ensureIndexHandler (fixes #9234) (#9235)
Towards the end of the function f.folderCfgs and f.folderRunners are
read without the lock.
2023-11-20 08:12:25 +01:00
Jakob Borg
a64ae36bcc
lib/model: Verify versioning on configuration reload (fixes #9106) (#9154) 2023-10-07 04:09:51 +02:00
Jakob Borg
051cbdc713
lib/fs, lib/model: Be careful about potentially negative durations (fixes #9112) (#9113)
I don't really understand under what circumstances, but sometimes these
calls panic with a "panic: counter cannot decrease in value" because the
value passed to Add() was negative.
2023-09-20 09:04:47 +02:00
Jakob Borg
c6334e61aa
all: Support multiple device connections (fixes #141) (#8918)
This adds the ability to have multiple concurrent connections to a single device. This is primarily useful when the network has multiple physical links for aggregated bandwidth. A single connection will never see a higher rate than a single link can give, but multiple connections are load-balanced over multiple links.

It is also incidentally useful for older multi-core CPUs, where bandwidth could be limited by the TLS performance of a single CPU core -- using multiple connections achieves concurrency in the required crypto calculations...

Co-authored-by: Simon Frei <freisim93@gmail.com>
Co-authored-by: tomasz1986 <twilczynski@naver.com>
Co-authored-by: bt90 <btom1990@googlemail.com>
2023-09-06 12:52:01 +02:00
Jakob Borg
5118538179
lib/model: Refactor folderRunners to use a serviceMap (#9071)
Instead of separately tracking the token.

Also changes serviceMap to have a channel version of RemoveAndWait, so
that it's possible to do the removal under a lock but wait outside of
the lock. And changed where we do that in connection close, reversing
the change that happened when I added the serviceMap in 40b3b9ad1.
2023-09-02 16:42:46 +02:00
Jakob Borg
acd767b30b
all: Remove lib/util package (#9049)
Grab-bag packages are nasty, this cleans it up a little by splitting it
into topical packages sempahore, netutil, stringutil, structutil.
2023-08-21 19:44:33 +02:00
Jakob Borg
40b3b9ad15
lib/model: Clean up index handler life cycle (fixes #9021) (#9038)
Co-authored-by: Simon Frei <freisim93@gmail.com>
2023-08-21 18:39:13 +02:00
André Colomb
a8cacdca94
lib/versioner: Minor fixes in comments and error message (#9031)
* lib/versioner: Factor out DefaultPath constant.

Replace several instances where .stversions is named literally to all
use the same definition in the versioner package.  Exceptions are the
packages where a cyclic dependency on versioner is impossible, or some
tests which combine the versions base path with other components.

* lib/versioner: Fix comment about trash can in simple versioner.

* lib/versioner: Fix wrong versioning type string in error message.

The error message shows the folder type instead of the versioning
type, although the correct field is used in the comparison.
2023-08-09 07:10:06 +00:00
Jakob Borg
8b87cd5229
lib/model: Reinstate setting folder idle state (#9029) 2023-08-08 07:24:02 +02:00
Jakob Borg
b9c08d3814
all: Add Prometheus-style metrics to expose some internal performance counters (fixes #5175) (#9003) 2023-08-04 19:57:30 +02:00
Jakob Borg
9d21b91124
all: Refactor the protocol/model interface a bit (ref #8981) (#9007) 2023-07-29 10:24:44 +02:00
Christian Kujau
6b6b2c6194
lib/model: use WARN for "Unexpected folder" messages (#8998) 2023-07-22 21:17:32 +00:00
deepsource-autofix[bot]
21c074cc2c
all: replace empty slice literal with var (#8990)
refactor: replace empty slice literal with `var`

An empty slice can be represented by `nil` or an empty slice literal. They are
functionally equivalent — their `len` and `cap` are both zero — but the `nil`
slice is the preferred style. For more information about empty slices,
see [Declaring Empty Slices](https://github.com/golang/go/wiki/CodeReviewComments#declaring-empty-slices).

Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>
2023-07-18 14:44:37 +00:00
deepsource-autofix[bot]
f23c41221b
all: fix unused method receiver (#8988)
refactor: fix unused method receiver

Methods with unused receivers can be a symptom of unfinished refactoring or a bug. To keep 
the same method signature, omit the receiver name or '_' as it is unused.

Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>
2023-07-18 14:34:50 +00:00
guangwu
27aba3567b
all: Minor staticcheck fixes (#8939) 2023-06-19 06:50:53 +00:00
Jakob Borg
81a4b22d43
lib/model: Improve test for unignored parent directories (#8926) 2023-06-04 15:32:03 +02:00
Anthony Goeckner
405cdedcd3
lib/model: Set platform data for new folders w/ ignorePerms (ref #8883) (#8907)
* Platform data (ownership, xattrs, etc.) is now set correctly for newly-received folders, even if the received folder has the NoPermissions flag.

* Call setPlatformData on receivers that have ignorePerms set to true.
2023-05-17 09:06:50 +02:00
Jakob Borg
1103a27337 all: Grand test refactor (fixes #8779, fixes #8799)
This fixes various test issues with Go 1.20.

- Most tests rewritten to use fakefs where possible
- Some tests that were already skipped, or dubious (invasive,
  unmaintainable, unclear what they even tested) have been removed
- Some actual code rewritten to better support testing in fakefs

Co-authored-by: Eric P <eric@kastelo.net>
2023-05-09 10:01:57 +00:00
Anthony Goeckner
7e31ec5417
lib/model: Set platform data, incl. copying ownership, for new folders w/ NoPermissions flag (#8883)
Platform data (ownership, xattrs, etc.) is now set correctly for newly-received folders, even if the received folder has the NoPermissions flag.
2023-05-02 11:11:39 +02:00
Jakob Borg
f59ffc8ddd
lib/model: Improve path generation for auto accepted folders (fixes #8859) (#8860)
- Make sure we don't try to use empty last path components
- Create the directory to "reserve" it once we've decided to use it
2023-04-11 13:07:22 +02:00
Simon Frei
3a41d4afa5
lib/model: Set enc. trailer size on pull (ref #8563, #8556) (#8839)
In the original fix in #8563 I simply forgot this. Which meant #8556
wasn't actually fixed, as the trialer size would have been 0 (default),
and thus we would have still sent the inflated size to encrypted peers.
2023-03-28 22:47:24 +02:00
Simon Frei
6a66aee489
lib/model: Fix file size inconsistency due to enc. trailer (#8840)
lib/model: Fix file size inconsisency due to enc. trailer

Fixes a regression due to PR #8563, while arguable the bug was actually
introduced in a much older PR #7155, but didn't have any bad effects so
far:
We account for the encryption trailer in the db updater routine,
calculating the file-info size there. However there's no guarantee that
the file-info at this point is still the exact same as when it was
written. It was before, but isn't anymore since introducing the new
EncryptedTrailerSize field.
Fix: Adjust the size in the info at the same place where the trailer is
written, i.e. we definitely have the actual size on disk.
2023-03-28 22:02:59 +02:00
Dimitri Papadopoulos Orfanos
526e21ae26
all: Fix typos found by codespell (#8833) 2023-03-21 08:07:28 +01:00
Jakob Borg
466b56ded1
lib/protocol: Cache expensive key operations (fixes #8599) (#8820)
This adds a cache to the expensive key generation operations. It's fixes
size LRU/MRU stuff to keep memory usage bounded under absurd conditions.

Also closes #8600.
2023-03-12 20:06:59 +01:00
Simon Frei
da72df6ffc
lib: Correctly handle encrypted trailer size (fixes #8556) (#8563) 2023-03-10 14:14:14 +01:00
Jakob Borg
8991ecf444 build: Add more GitHub Actions 2023-02-22 10:56:55 +01:00
greatroar
38f2b34d29
all: Use new Go 1.19 atomic types (#8772) 2023-02-07 12:07:34 +01:00
Simon Frei
d157d12037
lib/model: Only log at info level if setting change time fails (#8725) 2022-12-21 21:58:35 +01:00
Aleksey Vasenev
f9d68474ac
lib/model: Don't lower rescan interval from default on auto accepted enc folder (fixes #8572) (#8573) 2022-12-21 19:52:22 +01:00
Aleksey Vasenev
05738001ac
lib/model: Fixed adding empty items to device list (fixes #8646) (#8647) 2022-12-06 21:22:35 +01:00
André Colomb
ab0eb909a2
gui, lib/connections: Let the backend decide whether connection is local (fixes #8686) (#8694)
* lib/connections: Cache isLAN decision for later external access.

The check whether a remote device's address is on a local network
currently happens when handling the Hello message, to configure the
limiters.  Save the result to the ConnectionInfo and pass it out as
part of the model's ConnectionInfo struct in ConnectionStats().

* gui: Use provided connection attribute to distinguish LAN / WAN.

Replace the dumb IP address check which didn't catch common cases and
actually could contradict what the backend decided.  That could have
been confusing if the GUI says WAN, but the limiter is not actually
applied because the backend thinks it's a LAN.

Add strings for QUIC and relay connections to also differentiate
between LAN and WAN.

* gui: Redefine reception level icons for all connection types.

Move the mapping to the JS code, as it is much easier to handle
multiple switch cases by fall-through there.

QUIC is regarded no less than TCP anymore.  LAN and WAN make the
difference between levels 4 / 3 and 2 / 1:

{TCP,QUIC} LAN --> {TCP,QUIC} WAN --> Relay LAN --> Relay WAN -->
Disconnected.
2022-11-28 09:28:33 +01:00
Jakob Borg
6aa04118a6
lib/model: Correctly set xattrs on temp files (fixes #8667) (#8670) 2022-11-11 11:49:15 +01:00
Jakob Borg
a523fef78e
lib/model: Correctly handle xattrs on directories (fixes #8657) (#8658) 2022-11-09 06:54:04 +01:00
Jakob Borg
d3f50637d2
lib/model, lib/protocol: Fix file comparisons (fixes #8594) (#8603) 2022-10-16 17:04:28 +02:00
Simon Frei
a0c80e030a
lib/model: Fix warning log statement (ref #8583) (#8584) 2022-10-05 18:45:37 +02:00
Jakob Borg
6cac308bcd
all: Support syncing extended attributes (fixes #2698) (#8513)
This adds support for syncing extended attributes on supported
filesystem on Linux, macOS, FreeBSD and NetBSD. Windows is currently
excluded because the APIs seem onerous and annoying and frankly the uses
cases seem few and far between. On Unixes this also covers ACLs as those
are stored as extended attributes.

Similar to ownership syncing this will optional & opt-in, which two
settings controlling the main behavior: one to "sync" xattrs (read &
write) and another one to "scan" xattrs (only read them so other devices
can "sync" them, but not apply any locally).

Co-authored-by: Tomasz Wilczyński <twilczynski@naver.com>
2022-09-14 09:50:55 +02:00
luzpaz
837ffcfab5
all: Fix various user-facing and non-user-facing typos (#8509)
Found via `codespell -q 3 -S lang,./gui/default/vendor -L benchs,bu,inflight,ro`
2022-08-23 15:44:11 +02:00
Jakob Borg
b10d106a55
all: Modernize error wrapping (#8491)
This replaces old style errors.Wrap with modern fmt.Errorf and removes
the (direct) dependency on github.com/pkg/errors. A couple of cases are
adjusted by hand as previously errors.Wrap(nil, ...) would return nil,
which is not what fmt.Errorf does.
2022-08-16 10:01:49 +02:00
Jakob Borg
06273875ae
all: Make scanning ownership opt-in (#8497) 2022-08-12 07:47:20 +02:00
André Colomb
4d4bfe8032
lib/model: Skip paused folders in aggregated completion loop (fixes #8219) (#8220)
Locally paused folders will fail on checkFolderRunningLocked() and
therefore abort the loop.  Avoid this by skipping paused folders
directly.

Co-authored-by: Jakob Borg <jakob@kastelo.net>
2022-08-10 08:50:19 +02:00
Jakob Borg
209e68c1ba
build: Update quic-go for Go 1.19 (#8483)
Also adds idle time and keepalive parameters because how this is
configured has changed in the new package version. The values are those
that seems like might already be default, if keep-alives were enabled,
which is not obvious from the doc comments.

Also, Go 1.19 gofmt reformatting of comments.
2022-08-03 15:43:26 +02:00
deepsource-autofix[bot]
8e3f1190d1
lib/model: Use bytes.Equal instead of converting to string (#8469)
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>
2022-07-28 20:00:07 +02:00