Commit Graph

256 Commits

Author SHA1 Message Date
Jakob Borg
bda4016109
lib/protocol: Refactor interface (#9375)
This is a refactor of the protocol/model interface to take the actual
message as the parameter, instead of the broken-out fields:

```diff
type Model interface {
        // An index was received from the peer device
-       Index(conn Connection, folder string, files []FileInfo) error
+       Index(conn Connection, idx *Index) error
        // An index update was received from the peer device
-       IndexUpdate(conn Connection, folder string, files []FileInfo) error
+       IndexUpdate(conn Connection, idxUp *IndexUpdate) error
        // A request was made by the peer device
-       Request(conn Connection, folder, name string, blockNo, size int32, offset int64, hash []byte, weakHash uint32, fromTemporary bool) (RequestResponse, error)
+       Request(conn Connection, req *Request) (RequestResponse, error)
        // A cluster configuration message was received
-       ClusterConfig(conn Connection, config ClusterConfig) error
+       ClusterConfig(conn Connection, config *ClusterConfig) error
        // The peer device closed the connection or an error occurred
        Closed(conn Connection, err error)
        // The peer device sent progress updates for the files it is currently downloading
-       DownloadProgress(conn Connection, folder string, updates []FileDownloadProgressUpdate) error
+       DownloadProgress(conn Connection, p *DownloadProgress) error
 }
```

(and changing the `ClusterConfig` to `*ClusterConfig` for symmetry;
we'll be forced to use all pointers everywhere at some point anyway...)

The reason for this is that I have another thing cooking which is a
small troubleshooting change to check index consistency during transfer.
This required adding a field or two to the index/indexupdate messages,
and plumbing the extra parameters in umpteen changes is almost as big a
diff as this is. I figured let's do it once and avoid having to do that
in the future again...

The rest of the diff falls out of the change above, much of it being in
test code where we run these methods manually...
2024-01-31 08:18:27 +01:00
Jakob Borg
935a28c961
lib/model: Use a single lock (phase two: cleanup) (#9276)
Cleanup after #9275.

This renames `fmut` -> `mut`, removes the deadlock detector and
associated plumbing, renames some things from `...PRLocked` to
`...RLocked` and similar, and updates comments.

Apart from the removal of the deadlock detection machinery, no
functional code changes... i.e. almost 100% diff noise, have fun
reviewing.
2023-12-11 22:06:45 +01:00
Jakob Borg
6f1023665c
lib/model: Use a single lock (#9275)
I'm tired of the fmut/pmut shenanigans. This consolidates both under one
lock; I'm not convinced there are any significant performance
differences with this approach since we're literally just protecting map
juggling...

- The locking goes away when we were already under an appropriate fmut
lock.
- Where we had fmut.RLock()+pmut.Lock() it gets upgraded to an
fmut.Lock().
- Otherwise s/pmut/fmut/.

In order to avoid diff noise for an important change I did not do the
following cleanups, which will be filed in a PR after this one, if
accepted:

- Renaming fmut to just mut
- Renaming methods that refer to being "PRLocked" and stuff like that
- Removing the no longer relevant deadlock detector
- Comments referring to pmut and locking sequences...
2023-12-11 21:26:23 +01:00
Jakob Borg
c6334e61aa
all: Support multiple device connections (fixes #141) (#8918)
This adds the ability to have multiple concurrent connections to a single device. This is primarily useful when the network has multiple physical links for aggregated bandwidth. A single connection will never see a higher rate than a single link can give, but multiple connections are load-balanced over multiple links.

It is also incidentally useful for older multi-core CPUs, where bandwidth could be limited by the TLS performance of a single CPU core -- using multiple connections achieves concurrency in the required crypto calculations...

Co-authored-by: Simon Frei <freisim93@gmail.com>
Co-authored-by: tomasz1986 <twilczynski@naver.com>
Co-authored-by: bt90 <btom1990@googlemail.com>
2023-09-06 12:52:01 +02:00
Jakob Borg
5118538179
lib/model: Refactor folderRunners to use a serviceMap (#9071)
Instead of separately tracking the token.

Also changes serviceMap to have a channel version of RemoveAndWait, so
that it's possible to do the removal under a lock but wait outside of
the lock. And changed where we do that in connection close, reversing
the change that happened when I added the serviceMap in 40b3b9ad1.
2023-09-02 16:42:46 +02:00
Jakob Borg
acd767b30b
all: Remove lib/util package (#9049)
Grab-bag packages are nasty, this cleans it up a little by splitting it
into topical packages sempahore, netutil, stringutil, structutil.
2023-08-21 19:44:33 +02:00
Jakob Borg
40b3b9ad15
lib/model: Clean up index handler life cycle (fixes #9021) (#9038)
Co-authored-by: Simon Frei <freisim93@gmail.com>
2023-08-21 18:39:13 +02:00
André Colomb
a8cacdca94
lib/versioner: Minor fixes in comments and error message (#9031)
* lib/versioner: Factor out DefaultPath constant.

Replace several instances where .stversions is named literally to all
use the same definition in the versioner package.  Exceptions are the
packages where a cyclic dependency on versioner is impossible, or some
tests which combine the versions base path with other components.

* lib/versioner: Fix comment about trash can in simple versioner.

* lib/versioner: Fix wrong versioning type string in error message.

The error message shows the folder type instead of the versioning
type, although the correct field is used in the comparison.
2023-08-09 07:10:06 +00:00
Jakob Borg
9d21b91124
all: Refactor the protocol/model interface a bit (ref #8981) (#9007) 2023-07-29 10:24:44 +02:00
deepsource-autofix[bot]
21c074cc2c
all: replace empty slice literal with var (#8990)
refactor: replace empty slice literal with `var`

An empty slice can be represented by `nil` or an empty slice literal. They are
functionally equivalent — their `len` and `cap` are both zero — but the `nil`
slice is the preferred style. For more information about empty slices,
see [Declaring Empty Slices](https://github.com/golang/go/wiki/CodeReviewComments#declaring-empty-slices).

Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>
2023-07-18 14:44:37 +00:00
Jakob Borg
81a4b22d43
lib/model: Improve test for unignored parent directories (#8926) 2023-06-04 15:32:03 +02:00
Jakob Borg
1103a27337 all: Grand test refactor (fixes #8779, fixes #8799)
This fixes various test issues with Go 1.20.

- Most tests rewritten to use fakefs where possible
- Some tests that were already skipped, or dubious (invasive,
  unmaintainable, unclear what they even tested) have been removed
- Some actual code rewritten to better support testing in fakefs

Co-authored-by: Eric P <eric@kastelo.net>
2023-05-09 10:01:57 +00:00
Dimitri Papadopoulos Orfanos
526e21ae26
all: Fix typos found by codespell (#8833) 2023-03-21 08:07:28 +01:00
Jakob Borg
466b56ded1
lib/protocol: Cache expensive key operations (fixes #8599) (#8820)
This adds a cache to the expensive key generation operations. It's fixes
size LRU/MRU stuff to keep memory usage bounded under absurd conditions.

Also closes #8600.
2023-03-12 20:06:59 +01:00
greatroar
38f2b34d29
all: Use new Go 1.19 atomic types (#8772) 2023-02-07 12:07:34 +01:00
Jakob Borg
6cac308bcd
all: Support syncing extended attributes (fixes #2698) (#8513)
This adds support for syncing extended attributes on supported
filesystem on Linux, macOS, FreeBSD and NetBSD. Windows is currently
excluded because the APIs seem onerous and annoying and frankly the uses
cases seem few and far between. On Unixes this also covers ACLs as those
are stored as extended attributes.

Similar to ownership syncing this will optional & opt-in, which two
settings controlling the main behavior: one to "sync" xattrs (read &
write) and another one to "scan" xattrs (only read them so other devices
can "sync" them, but not apply any locally).

Co-authored-by: Tomasz Wilczyński <twilczynski@naver.com>
2022-09-14 09:50:55 +02:00
Jakob Borg
b10d106a55
all: Modernize error wrapping (#8491)
This replaces old style errors.Wrap with modern fmt.Errorf and removes
the (direct) dependency on github.com/pkg/errors. A couple of cases are
adjusted by hand as previously errors.Wrap(nil, ...) would return nil,
which is not what fmt.Errorf does.
2022-08-16 10:01:49 +02:00
Jakob Borg
a3c724f2c3
all: Add build constants for runtime.GOOS comparisons (#8442)
all: Add package runtimeos for runtime.GOOS comparisons

I grew tired of hand written string comparisons. This adds generated
constants for the GOOS values, and predefined Is$OS constants that can
be iffed on. In a couple of places I rewrote trivial switch:es to if:s,
and added Illumos where we checked for Solaris (because they are
effectively the same, and if we're going to target one of them that
would be Illumos...).
2022-07-28 19:36:39 +02:00
André Colomb
1eda82b95f
lib/model: Improve remoteFolderState reporting (fixes #8266) (#8283) 2022-04-22 08:42:20 +02:00
Eng Zer Jun
bc27aa12cd
all: use T.TempDir to create temporary test directory (#8280)
This commit replaces `os.MkdirTemp` with `t.TempDir` in tests. The
directory created by `t.TempDir` is automatically removed when the test
and all its subtests complete.

Prior to this commit, temporary directory created using `os.MkdirTemp`
needs to be removed manually by calling `os.RemoveAll`, which is omitted
in some tests. The error handling boilerplate e.g.
	defer func() {
		if err := os.RemoveAll(dir); err != nil {
			t.Fatal(err)
		}
	}
is also tedious, but `t.TempDir` handles this for us nicely.

Reference: https://pkg.go.dev/testing#T.TempDir
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2022-04-15 07:44:06 +04:00
André Colomb
0c46e0a9cc
gui, lib/model: Mark folders unaccepted by remote device (fixes #8202) (#8201) 2022-04-10 22:47:57 +02:00
Simon Frei
db72579f0e
lib: Get rid of buggy filesystem wrapping (#8257) 2022-04-10 20:55:05 +02:00
Simon Frei
635085d139
lib/db, lib/model: Remove filesystem state from FileSet (fixes #7850) (#8151) 2022-01-31 10:12:52 +01:00
Simon Frei
21d04b895a
lib, gui: Default ignores for new folders (fixes #7428) (#7530) 2022-01-13 23:38:21 +01:00
Jakob Borg
4b750b6dc3
all: Remove usage of deprecated io/ioutil (#7971)
As of Go 1.16 io/ioutil is deprecated. This replaces usage with the
corresponding functions in package os and package io.
2021-11-22 08:59:47 +01:00
Simon Frei
591e4d8af1
gui, lib: Fix tracking deleted locally-changed on encrypted (fixes #7715) (#7726) 2021-11-10 09:46:21 +01:00
Simon Frei
c78fa42f31
lib/connections: Dial devices in parallel (#7783) 2021-06-25 11:38:04 +02:00
Simon Frei
857caf3637
lib/connections: Trigger dialer when connection gets closed (#7753) 2021-06-17 13:57:44 +02:00
Simon Frei
18592af993
lib/model: Fix wrongly hardcoded arguments in test helper (#7749) 2021-06-05 17:01:23 +02:00
Simon Frei
41baccb85d
lib/model: Fix passwords on receive-enc needing token (ref #7518) (#7739) 2021-06-03 15:39:49 +02:00
Simon Frei
004eded398
lib/model: Don't share with introduced device if encrypted (fixes #7724) (#7734) 2021-06-03 15:02:57 +02:00
Simon Frei
df48276300
lib/model: Ensure indexes are only received after checking IDs (ref #7649) (#7689) 2021-06-03 14:58:50 +02:00
Simon Frei
5b90a98650
lib/model: Fix addFakeConn and other test improvements (#7684) 2021-05-16 17:23:27 +02:00
Simon Frei
5cb2a10138
lib/model: Improve encryption cluster-config errors (#7658) 2021-05-11 07:55:44 +02:00
Simon Frei
ec0a66c75b
lib/db, lib/model: Refactor removing expired pending folders (#7537) 2021-04-11 15:24:08 +02:00
Simon Frei
0d7a77ba85
lib/model: Reset conn when enc token is missing (fixes #7198) (#7518) 2021-03-23 10:38:40 +01:00
Simon Frei
924b96856f
lib: Handle adding enc folders on an existing conn (fixes #7509) (#7510) 2021-03-22 21:50:19 +01:00
Lars Lehtonen
a87c5515bd
lib/model: Error check in test (#7508) 2021-03-21 10:32:17 +01:00
Simon Frei
310fba4c12
lib: Return error from db.FileSet.Snapshot (fixes #7419, ref #5907) (#7424) 2021-03-07 13:43:22 +01:00
Simon Frei
3d91f7c975
lib: Use counterfeiter to mock interfaces in tests (#7375) 2021-03-03 08:53:50 +01:00
Simon Frei
80fc238bec
all: Automatic/disabled folder-config when receive-encrypted (#7327) 2021-02-12 22:51:29 +01:00
Simon Frei
7e4e2f3720
lib/versioner: Improve error messages (fixes #7354) (#7357) 2021-02-12 20:30:51 +01:00
Simon Frei
d842197931
lib/model: Disable tests involving scrypt with -short (fixes #7344) (#7346) 2021-02-08 17:13:28 +01:00
Simon Frei
ffc14a77c6
all: Add configurable defaults (fixes #4224, fixes #6086) (#7131) 2021-02-04 21:10:41 +01:00
Audrius Butkevicius
a7d9268e4d
lib/model: Make /browse endpoint return sane objects (#7306) 2021-02-01 09:27:34 +01:00
Simon Frei
f63cdbfcfa
lib: Apply config changes sequentially (ref #5298) (#7188) 2021-01-15 15:43:34 +01:00
Jakob Borg
253049a4cc
lib/api, lib/model: Avoid contention on filesystem for DB status call (ref #7270) (#7271)
This splits the ignore getting to two methods, one that loads from disk
(the old one) and one that just returns whatever is already loaded (the
new one). The folder summary service which is just interested in stats
now uses the latter method. This means that it, and API calls that call
it, does not get blocked by folder I/O.
2021-01-12 16:25:21 +01:00
Jakob Borg
0b193b76c2
lib/config, lib/connections: Add optional connection limits (fixes #7176) (#7223)
This adds two new configuration options:

    // The number of connections at which we stop trying to connect to more
    // devices, zero meaning no limit. Does not affect incoming connections.
    ConnectionLimitEnough int

    // The maximum number of connections which we will allow in total, zero
    // meaning no limit. Affects incoming connections and prevents
    // attempting outgoing connections.
    ConnectionLimitMax int

These can be used to limit the number of concurrent connections in
various ways.
2021-01-11 15:14:44 +01:00
Simon Frei
78bd0341a8
all: Handle errors opening db/creating file-set (ref #5907) (#7150) 2020-12-21 12:59:22 +01:00
Simon Frei
c845e245a1
lib: Close underlying conn in protocol (fixes #7165) (#7212) 2020-12-21 11:40:51 +01:00