Commit Graph

74 Commits

Author SHA1 Message Date
Jakob Borg
bda4016109
lib/protocol: Refactor interface (#9375)
This is a refactor of the protocol/model interface to take the actual
message as the parameter, instead of the broken-out fields:

```diff
type Model interface {
        // An index was received from the peer device
-       Index(conn Connection, folder string, files []FileInfo) error
+       Index(conn Connection, idx *Index) error
        // An index update was received from the peer device
-       IndexUpdate(conn Connection, folder string, files []FileInfo) error
+       IndexUpdate(conn Connection, idxUp *IndexUpdate) error
        // A request was made by the peer device
-       Request(conn Connection, folder, name string, blockNo, size int32, offset int64, hash []byte, weakHash uint32, fromTemporary bool) (RequestResponse, error)
+       Request(conn Connection, req *Request) (RequestResponse, error)
        // A cluster configuration message was received
-       ClusterConfig(conn Connection, config ClusterConfig) error
+       ClusterConfig(conn Connection, config *ClusterConfig) error
        // The peer device closed the connection or an error occurred
        Closed(conn Connection, err error)
        // The peer device sent progress updates for the files it is currently downloading
-       DownloadProgress(conn Connection, folder string, updates []FileDownloadProgressUpdate) error
+       DownloadProgress(conn Connection, p *DownloadProgress) error
 }
```

(and changing the `ClusterConfig` to `*ClusterConfig` for symmetry;
we'll be forced to use all pointers everywhere at some point anyway...)

The reason for this is that I have another thing cooking which is a
small troubleshooting change to check index consistency during transfer.
This required adding a field or two to the index/indexupdate messages,
and plumbing the extra parameters in umpteen changes is almost as big a
diff as this is. I figured let's do it once and avoid having to do that
in the future again...

The rest of the diff falls out of the change above, much of it being in
test code where we run these methods manually...
2024-01-31 08:18:27 +01:00
Jakob Borg
5118538179
lib/model: Refactor folderRunners to use a serviceMap (#9071)
Instead of separately tracking the token.

Also changes serviceMap to have a channel version of RemoveAndWait, so
that it's possible to do the removal under a lock but wait outside of
the lock. And changed where we do that in connection close, reversing
the change that happened when I added the serviceMap in 40b3b9ad1.
2023-09-02 16:42:46 +02:00
Jakob Borg
b9c08d3814
all: Add Prometheus-style metrics to expose some internal performance counters (fixes #5175) (#9003) 2023-08-04 19:57:30 +02:00
Jakob Borg
9d21b91124
all: Refactor the protocol/model interface a bit (ref #8981) (#9007) 2023-07-29 10:24:44 +02:00
Jakob Borg
1103a27337 all: Grand test refactor (fixes #8779, fixes #8799)
This fixes various test issues with Go 1.20.

- Most tests rewritten to use fakefs where possible
- Some tests that were already skipped, or dubious (invasive,
  unmaintainable, unclear what they even tested) have been removed
- Some actual code rewritten to better support testing in fakefs

Co-authored-by: Eric P <eric@kastelo.net>
2023-05-09 10:01:57 +00:00
Jakob Borg
6cac308bcd
all: Support syncing extended attributes (fixes #2698) (#8513)
This adds support for syncing extended attributes on supported
filesystem on Linux, macOS, FreeBSD and NetBSD. Windows is currently
excluded because the APIs seem onerous and annoying and frankly the uses
cases seem few and far between. On Unixes this also covers ACLs as those
are stored as extended attributes.

Similar to ownership syncing this will optional & opt-in, which two
settings controlling the main behavior: one to "sync" xattrs (read &
write) and another one to "scan" xattrs (only read them so other devices
can "sync" them, but not apply any locally).

Co-authored-by: Tomasz Wilczyński <twilczynski@naver.com>
2022-09-14 09:50:55 +02:00
Jakob Borg
06273875ae
all: Make scanning ownership opt-in (#8497) 2022-08-12 07:47:20 +02:00
deepsource-autofix[bot]
8e3f1190d1
lib/model: Use bytes.Equal instead of converting to string (#8469)
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>
2022-07-28 20:00:07 +02:00
Jakob Borg
a3c724f2c3
all: Add build constants for runtime.GOOS comparisons (#8442)
all: Add package runtimeos for runtime.GOOS comparisons

I grew tired of hand written string comparisons. This adds generated
constants for the GOOS values, and predefined Is$OS constants that can
be iffed on. In a couple of places I rewrote trivial switch:es to if:s,
and added Illumos where we checked for Solaris (because they are
effectively the same, and if we're going to target one of them that
would be Illumos...).
2022-07-28 19:36:39 +02:00
Jakob Borg
5958f42294 lib/model: Clarify normal shallow copy 2022-07-28 17:41:07 +02:00
Jakob Borg
7bdb5faa9c
all: Remove or convert deprecated API usages (#8459) 2022-07-28 17:14:49 +02:00
Jakob Borg
adce6fa473
all: Support syncing ownership (fixes #1329) (#8434)
This adds support for syncing ownership on Unixes and on Windows. The
scanner always picks up ownership information, but it is not applied
unless the new folder option "Sync Ownership" is set.

Ownership data is stored in a new FileInfo field called "platform data". This
is intended to hold further platform-specific data in the future
(specifically, extended attributes), which is why the whole design is a
bit overkill for just ownership.
2022-07-26 08:24:58 +02:00
Eng Zer Jun
bc27aa12cd
all: use T.TempDir to create temporary test directory (#8280)
This commit replaces `os.MkdirTemp` with `t.TempDir` in tests. The
directory created by `t.TempDir` is automatically removed when the test
and all its subtests complete.

Prior to this commit, temporary directory created using `os.MkdirTemp`
needs to be removed manually by calling `os.RemoveAll`, which is omitted
in some tests. The error handling boilerplate e.g.
	defer func() {
		if err := os.RemoveAll(dir); err != nil {
			t.Fatal(err)
		}
	}
is also tedious, but `t.TempDir` handles this for us nicely.

Reference: https://pkg.go.dev/testing#T.TempDir
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2022-04-15 07:44:06 +04:00
Simon Frei
db72579f0e
lib: Get rid of buggy filesystem wrapping (#8257) 2022-04-10 20:55:05 +02:00
Simon Frei
635085d139
lib/db, lib/model: Remove filesystem state from FileSet (fixes #7850) (#8151) 2022-01-31 10:12:52 +01:00
Jakob Borg
4b750b6dc3
all: Remove usage of deprecated io/ioutil (#7971)
As of Go 1.16 io/ioutil is deprecated. This replaces usage with the
corresponding functions in package os and package io.
2021-11-22 08:59:47 +01:00
Simon Frei
591e4d8af1
gui, lib: Fix tracking deleted locally-changed on encrypted (fixes #7715) (#7726) 2021-11-10 09:46:21 +01:00
tomasz1986
5a1f6cb813
lib/fs: Improve case conflict error message (fixes #7827) (#7829) 2021-08-01 22:44:49 +02:00
Simon Frei
df48276300
lib/model: Ensure indexes are only received after checking IDs (ref #7649) (#7689) 2021-06-03 14:58:50 +02:00
Simon Frei
5b90a98650
lib/model: Fix addFakeConn and other test improvements (#7684) 2021-05-16 17:23:27 +02:00
Simon Frei
310fba4c12
lib: Return error from db.FileSet.Snapshot (fixes #7419, ref #5907) (#7424) 2021-03-07 13:43:22 +01:00
Simon Frei
ffc14a77c6
all: Add configurable defaults (fixes #4224, fixes #6086) (#7131) 2021-02-04 21:10:41 +01:00
Simon Frei
f63cdbfcfa
lib: Apply config changes sequentially (ref #5298) (#7188) 2021-01-15 15:43:34 +01:00
Simon Frei
c48eb4241a
lib/model: Fix child-check when deleting dirs in pull (#7236) 2021-01-02 21:40:37 +01:00
Simon Frei
a05dc6cc47
lib/model: Cleanup redundant filesystem variables in folders (#7237) 2020-12-27 22:26:25 +01:00
Simon Frei
78bd0341a8
all: Handle errors opening db/creating file-set (ref #5907) (#7150) 2020-12-21 12:59:22 +01:00
Simon Frei
9524b51708
all: Implement suture v4-api (#6947) 2020-11-17 13:19:04 +01:00
Simon Frei
cc9ea9db89
lib/folder: Clear pull errors when nothing is needed anymore (#7093) 2020-11-06 14:22:20 +01:00
Audrius Butkevicius
e027175446
all: Move remaining protos to use the vanity plugin (#7009) 2020-10-02 08:07:05 +02:00
Simon Frei
8210466b03
lib/model: Consider case conflicts when checking to be deleted items (#6986) 2020-09-12 07:45:50 +02:00
Simon Frei
c5c23ed10f
lib/model: Consider existing file when handling symlink on windows (#6977) 2020-09-10 10:52:38 +02:00
Simon Frei
56d48d341f
lib/model: Fix case-only renames on pull (#6978) 2020-09-10 08:23:54 +02:00
Simon Frei
e3cd9219b8
lib/model: Don't fail over case-conflict on tempfile (fixes #6973) (#6975) 2020-09-09 11:47:14 +02:00
Audrius Butkevicius
e9bb17307d
lib/fs: Unwrap mtimeFile, get fd the "correct" way (ref #6875) (#6877) 2020-08-07 07:47:48 +02:00
Audrius Butkevicius
96e197e502
lib/model: Don't close file early (fixes #6875) (#6876) 2020-08-03 21:54:42 +02:00
Simon Frei
932d8c69de lib/fs: Properly handle case insensitive systems (fixes #1787, fixes #2739, fixes #5708)
With this change we emulate a case sensitive filesystem on top of
insensitive filesystems. This means we correctly pick up case-only renames
and throw a case conflict error when there would be multiple files differing
only in case.

This safety check has a small performance hit (about 20% more filesystem
operations when scanning for changes). The new advanced folder option
`caseSensitiveFS` can be used to disable the safety checks, retaining the
previous behavior on systems known to be fully case sensitive.

Co-authored-by: Jakob Borg <jakob@kastelo.net>
2020-07-28 11:15:11 +02:00
Audrius Butkevicius
4812fd3ec1
all: Add copy-on-write filesystem support (fixes #4271) (#6746) 2020-06-18 08:15:47 +02:00
Simon Frei
6976219d6d
lib/model: Check dir before deletion when pulling (#6741) 2020-06-16 15:20:08 +02:00
Simon Frei
c3b5eba205
lib/model: Fix checking children when trying to delete a dir (fixes #6646) (#6647) 2020-05-25 08:46:24 +02:00
Audrius Butkevicius
a1c5b44c74
lib/model: Fix rename handling (ref #6650) (#6652) 2020-05-16 14:39:27 +02:00
Simon Frei
b5fc332782
lib/model: Merge add and start folder funcs and related refactor (#6594) 2020-05-06 08:34:54 +02:00
Audrius Butkevicius
782bd08aad
lib/model: Add option to disable fsync (#6588)
* lib/model: Add option to disable fsync

* Fix test

* Dont open stuff for no reason
2020-05-01 08:36:46 +01:00
Jakob Borg
6c73617974
lib/model: Use semaphore to limit concurrent folder writes (fixes #6541) (#6573) 2020-04-27 00:13:18 +02:00
Simon Frei
d3ed4de4ed
lib/model: Don't exit pullerRoutine on cancelled ctx (fixes #6559) (#6562)
* lib/model: Don't exit pullerRoutine on cancelled ctx (fixes #6559)

* actual fix
2020-04-21 18:55:14 +01:00
Simon Frei
08f0e125ef all: Transactionalize db.FileSet (fixes #5952) (#6239) 2020-01-21 18:23:08 +01:00
Jakob Borg
c71116ee94
Implement database abstraction, error checking (ref #5907) (#6107)
This PR does two things, because one lead to the other:

- Move the leveldb specific stuff into a small "backend" package that
defines a backend interface and the leveldb implementation. This allows,
potentially, in the future, switching the db implementation so another
KV store should we wish to do so.

- Add proper error handling all along the way. The db and backend
packages are now errcheck clean. However, I drew the line at modifying
the FileSet API in order to keep this manageable and not continue
refactoring all of the rest of Syncthing. As such, the FileSet methods
still panic on database errors, except for the "database is closed"
error which is instead handled by silently returning as quickly as
possible, with the assumption that we're anyway "on the way out".
2019-11-29 09:11:52 +01:00
Simon Frei
5edf4660e2
lib/model: Prevent cleanup-race in testing (ref #6152) (#6155) 2019-11-14 23:08:40 +01:00
Simon Frei
f80ce17497
lib/model: In tests prevent goroutine leaks and increase timeouts (#6152) 2019-11-13 10:21:54 +01:00
Simon Frei
85e6a77f25 lib/model: Remove some testing deadlocks (#6138) 2019-11-08 18:53:51 +01:00
Simon Frei
bee7cce081
lib/model: Add folders on start in model (#6135) 2019-11-08 10:56:16 +01:00