Commit Graph

1862 Commits

Author SHA1 Message Date
Simon Frei
ffc14a77c6
all: Add configurable defaults (fixes #4224, fixes #6086) (#7131) 2021-02-04 21:10:41 +01:00
greatroar
31119ed61a
lib/ignore: Store cache timestamps as Unix ns counts (#7326) 2021-02-04 18:39:06 +01:00
Simon Frei
070bf3b776
lib/db: Report number of repaired items from checkGlobal (#7329) 2021-02-04 14:42:46 +01:00
greatroar
42917d707d
lib/scanner: Remove unused field, move WaitGroup.Add out of loop (#7323) 2021-02-03 14:25:24 +01:00
Jakob Borg
3b7a57d108
lib/protocol: Hide repeated data blocks in a given file (#7319) 2021-02-02 20:15:14 +01:00
Audrius Butkevicius
a7d9268e4d
lib/model: Make /browse endpoint return sane objects (#7306) 2021-02-01 09:27:34 +01:00
Simon Frei
052dc13487
lib/model: Correct pull progress for small files (fixes #7263) (#7316) 2021-01-31 23:40:15 +01:00
Simon Frei
249bcb3a01
lib/model: Optimize rename detection in scanner (#7315) 2021-01-31 21:02:42 +01:00
greatroar
fbe52faf49
lib/scanner: Allocate structure for final partial block (#7310)
Benchmark results on Linux/amd64, using updated benchmark for old and
new:

name        old time/op    new time/op    delta
HashFile-8    88.6ms ± 1%    88.3ms ± 1%   -0.33%  (p=0.046 n=19+19)

name        old speed      new speed      delta
HashFile-8   201MB/s ± 1%   202MB/s ± 1%   +0.33%  (p=0.044 n=19+19)

name        old alloc/op   new alloc/op   delta
HashFile-8    59.4kB ± 0%    46.1kB ± 0%  -22.47%  (p=0.000 n=14+20)

name        old allocs/op  new allocs/op  delta
HashFile-8      29.0 ± 0%      27.0 ± 0%   -6.90%  (p=0.000 n=20+20)

Co-authored-by: greatroar <@>
2021-01-28 14:23:24 +01:00
greatroar
8b86171642
lib/stun: Inline util.OnDone, comment on its purpose (#7308)
Co-authored-by: greatroar <@>
2021-01-27 19:27:00 +01:00
Simon Frei
e19d6e993d
lib/fs: Cache all real-case results (fixes #7270) (#7286) 2021-01-27 19:25:34 +01:00
greatroar
ef0473c091
lib/util, lib/svcutil: Remove unused code (#7309)
Duplicates the definition in lib/svcutil.

Co-authored-by: greatroar <@>
2021-01-27 16:33:01 +01:00
greatroar
6c3e187d1d
lib/svcutil: Simplify doneService (#7303)
OnSupervisorDone no longer allocates.

Co-authored-by: greatroar <@>
2021-01-25 16:27:17 +01:00
André Colomb
e32a516b5f
lib/model: Forget pending folders no longer announced in ClusterConfig (fixes #5187) (#7205)
* lib/db: Add ExpirePendingFolders().

Use-case is to drop any no-longer-pending folders for a specific
device when parsing its ClusterConfig message where previously offered
folders are not mentioned any more.

The timestamp in ObservedFolder is stored with only second precision,
so round to seconds here as well.  This allows calling the function
within the same second of adding or updating entries.

* lib/model: Weed out pending folders when receiving ClusterConfig.

Filter the entries by timestamp, which must be newer than or equal to
the reception time of the ClusterConfig.  For just mentioned ones,
this assumption will hold as AddOrUpdatePendingFolder() updates the
timestamp.

* lib/model, gui: Notify when one or more pending folders expired.

Introduce new event type FolderOfferCancelled and use it to trigger a
complete refreshCluster() cycle.  Listing individual entries would be
much more code and probably just as much work to answer the API
request.

* lib/model: Add comment and rename ExpirePendingFolders().

* lib/events: Rename FolderOfferCancelled to ClusterPendingChanged.

* lib/model: Reuse ClusterPendingChanged event for cleanPending()

Changing the config does not necessarily mean that the
/resut/cluster/pending endpoints need to be refreshed, but only if
something was actually removed.  Detect this and indicate it through
the ClusterPendingChanged event, which is already hooked up to requery
respective endpoints within the GUI.

No more need for a separate refreshCluster() in reaction to
ConfigSaved event or calling refreshConfig().

* lib/model: Gofmt.

* lib/db: Warn instead of info log for failed removal.

* gui: Fix pending notifications not loading on GUI start.

* lib/db: Use short device ID in log message.

* lib/db: Return list of expired folder IDs after deleting them.

* lib/model: Refactor Pending...Changed events.

* lib/model: Adjust format of removed pending folders enumeration.

Use an array of objects with device / folder ID properties, matching
the other places where it's used.

* lib/db: Drop invalid entries in RemovePendingFoldersBeforeTime().

* lib/model: Gofmt.

My local gofmt did not complain here, strangely...

* gui: Handle PendingDevicesChanged event.

Even though it currently only holds one device at a time, wrap the
contents in an array under the "added" property name.

* lib/model: Fix null values in PendingFoldersChanged removed member.

* gui: Handle PendingFoldersChanged event.

* lib/model: Simplify construction of expiredPendingList.

* lib/model: Reduce code duplication in cleanPending().

Use goto and a label for the common parts of calling the DB removal
function and building the event data part.

* lib/events, gui: Mark ...Rejected events deprecated.

Extend comments explaining the conditions when the replacement event
types are emitted.

* lib/model: Wrap removed devices in array of objects as well.

* lib/db: Use iter.Value() instead of needless db.Get(iter.Key())

* lib/db: Add comment explaining RemovePendingFoldersBeforeTime().

* lib/model: Rename fields folderID and deviceID in event data.

* lib/db: Only list actually expired IDs as removed.

Skip entries where Delete() failed as well as invalid entries that got
removed automatically.

* lib/model: Gofmt
2021-01-25 10:58:10 +00:00
greatroar
6da83ac9f5
lib/util: Remove duplicate error handling code (#7299)
This is also in lib/svcutil, and never used by clients.

Co-authored-by: greatroar <@>
2021-01-24 20:19:10 +01:00
greatroar
9c88efd55f
lib/util: Don't modify input in UniqueTrimmedStrings (#7288)
Also clarified the comment.
2021-01-16 17:39:15 +01:00
Jakob Borg
ffcb57580f
cmd/syncthing: Provide early startup for config service (ref #7188) (#7285) 2021-01-16 12:58:02 +01:00
Simon Frei
f63cdbfcfa
lib: Apply config changes sequentially (ref #5298) (#7188) 2021-01-15 15:43:34 +01:00
Simon Frei
b2d82da20d
lib/model: Pull when folder leaves error state (fixes #7280) (#7281) 2021-01-14 13:29:01 +01:00
Jakob Borg
36acaf5e21
lib/fs: Avoid blocking new caseFs creation while waiting to drop cache (fixes #7273) (#7275) 2021-01-13 17:45:29 +01:00
Jakob Borg
253049a4cc
lib/api, lib/model: Avoid contention on filesystem for DB status call (ref #7270) (#7271)
This splits the ignore getting to two methods, one that loads from disk
(the old one) and one that just returns whatever is already loaded (the
new one). The folder summary service which is just interested in stats
now uses the latter method. This means that it, and API calls that call
it, does not get blocked by folder I/O.
2021-01-12 16:25:21 +01:00
Jakob Borg
8f199e12b3
lib/fs, lib/model: Reduce lock contention on NewCaseFilesystem (fixes #7268) (#7269) 2021-01-12 16:22:21 +01:00
greatroar
f6fac3e949
lib/ur: Plug file descriptor leak in Linux memorySize (#7266) 2021-01-11 15:15:21 +01:00
Jakob Borg
0b193b76c2
lib/config, lib/connections: Add optional connection limits (fixes #7176) (#7223)
This adds two new configuration options:

    // The number of connections at which we stop trying to connect to more
    // devices, zero meaning no limit. Does not affect incoming connections.
    ConnectionLimitEnough int

    // The maximum number of connections which we will allow in total, zero
    // meaning no limit. Affects incoming connections and prevents
    // attempting outgoing connections.
    ConnectionLimitMax int

These can be used to limit the number of concurrent connections in
various ways.
2021-01-11 15:14:44 +01:00
Jakob Borg
b13b15758d
lib/connections, lib/model: Track last connection duration (ref #7223) (#7242)
This adds a statistic to track the last connection duration per device.
It isn't used for much in this PR, but it's available for #7223 to use
in deciding how to order device connection attempts (deprioritizing
devices that just dropped our connection the last time).
2021-01-05 17:45:07 +01:00
Simon Frei
c48eb4241a
lib/model: Fix child-check when deleting dirs in pull (#7236) 2021-01-02 21:40:37 +01:00
Simon Frei
0f8290485e
lib/model: Handle index sender terminating due to error (fixes #7231) (#7232) 2020-12-30 09:59:11 +01:00
André Colomb
3d1edd2492
lib/fs: Fix TestChmodDir depending on umask (fixes #6551) (#7241)
The test would fail if the umask on UNIX is greater than 0022, because
the OS transparently subtracts it from the mode passed to Mkdir(), as
the Go documentation confirms.

Our goal here is not to test os.Mkdir(), so just make sure the desired
mode is actually set by forcing it afterwards.
2020-12-30 09:56:10 +01:00
Simon Frei
a05dc6cc47
lib/model: Cleanup redundant filesystem variables in folders (#7237) 2020-12-27 22:26:25 +01:00
Simon Frei
f13e6ca631
lib/model: Remove obsolete return val from ccHandleFolders (ref #6443) (#7229) 2020-12-23 13:10:08 +01:00
Simon Frei
a20a5f61f0
lib/ur: Send unreported failures on shutdown (#7164) 2020-12-22 20:17:14 +01:00
Jakob Borg
05f25e600e
lib/connections: Refactor connection loop (#7177)
This breaks out some methods from the connection loop to make it simpler
to manage and understand.

Some slight simplifications to remove the `seen` variable (we can filter
`nextDial` based on times are in the future or not, so we don't need to
track `seen`) and adding a minimum loop interval (5s) in case some
dialer goes haywire and requests a 0s redial interval or such.

Otherwise no significant behavioral changes.
2020-12-21 16:40:13 +01:00
Simon Frei
a744dee94c
lib/fs: Correct wrapping order for meaningful log-caller (#7209) 2020-12-21 13:01:34 +01:00
Simon Frei
78bd0341a8
all: Handle errors opening db/creating file-set (ref #5907) (#7150) 2020-12-21 12:59:22 +01:00
Simon Frei
c845e245a1
lib: Close underlying conn in protocol (fixes #7165) (#7212) 2020-12-21 11:40:51 +01:00
Simon Frei
4a787986cd
lib/db: Prevent IndexID creation race (#7211) 2020-12-21 11:32:59 +01:00
Simon Frei
bd0c9913cf
lib/db: Remove index ids when dropping folder (#7200) 2020-12-21 11:10:59 +01:00
Simon Frei
d904dfa191
lib/model: Fix flaky test and add some scanning debug (#7214) 2020-12-20 18:13:35 +01:00
Simon Frei
fa40ccece1
lib: Consistently set suture logging (#7202) 2020-12-18 19:44:00 +01:00
Simon Frei
7919310dc6
lib/model: Unflake TestIgnoreDeleteUnignore (#7208) 2020-12-18 18:42:09 +01:00
Simon Frei
739e99c4d9
lib/config: Remove deprecated pending entries from config (ref #6443) (#7204) 2020-12-17 22:49:29 +01:00
André Colomb
7502997e7e
all: Store pending devices and folders in database (fixes #7178) (#6443) 2020-12-17 19:54:31 +01:00
Jakob Borg
4142a431b5 model: Actually print folder description in "Overriding" log message 2020-12-12 12:32:24 +01:00
Simon Frei
0db3b7a530
build: Switch to gopsutil's v3 module (#7191) 2020-12-10 16:43:15 +01:00
Jakob Borg
ec5a5d5218
lib/api: Returns tags in version as list (#7190) 2020-12-10 12:22:09 +01:00
Jakob Borg
e9b68a224c
lib/connections: Handle QUIC not being available (#7186)
This does two things:

- Exclude QUIC from go1.16 builds, automatically, for now, since it
  doesn't work and just panics.

- Provide some fake listeners and dialers when QUIC is disabled.

These fake listeners and dialers indicate that they are disabled and
unsupported, which silences "Dialing $address: unknown address scheme:
quic" type of stuff which is not super helpful to the user.
2020-12-09 19:23:50 +01:00
Simon Frei
8fd6b1d428
lib/protocol: Handle slashified paths in IsEncryptedParent (fixes #7184) (#7187) 2020-12-09 18:16:14 +01:00
Simon Frei
cccbb0bd5e
lib/ur: Reset timer when there's nothing to report (#7169) 2020-11-28 20:09:22 +01:00
Simon Frei
240ae0c14f
lib/model: Unflake TestRequestReceiveEncryptedLocalNoSend (#7167) 2020-11-27 20:53:03 +01:00
Simon Frei
bbb22c8c80
lib/protocol: Send Close message on read error (#7141) 2020-11-27 11:31:20 +01:00
Simon Frei
a9764fc16c
lib: Skip deleted, locally changed on recv-enc folders (fixes #7153) (#7154) 2020-11-27 11:26:36 +01:00
Audrius Butkevicius
af13f0cd35
lib/stun: Don't notify about address changes if it's not useful (fixes #7144) (#7159) 2020-11-27 07:34:30 +01:00
Simon Frei
e1b958284e
lib/api: Shut the api down gracefully (fixes #7138) (#7157) 2020-11-26 15:49:39 +01:00
Simon Frei
bf7d03d029
lib/model: Fix enc file size when pulling (fixes #7152) (#7155) 2020-11-25 22:57:25 +01:00
Simon Frei
3169212046
lib/db: Do not reset index-id when dropping device (ref #7135) (#7156)
This reverts commit 641b7aee38.
2020-11-25 22:54:05 +01:00
Simon Frei
54b50e3d52
lib/fs, lib/model: Cover more windowsyness sanitizing paths (fixes #7075) (#7158) 2020-11-25 22:52:46 +01:00
André Colomb
e6595c1ab9
lib/model: Simplify access to Folder and Device configuration. (#7151)
Use the accessor methods with a device argument instead of looking up
in a temporary map of the same data.
2020-11-24 22:20:50 +01:00
Simon Frei
100ef10d84
lib/model: Don't send locally changed on recv-enc (fixes #7137) (#7147) 2020-11-24 21:49:45 +01:00
Simon Frei
5d2c83a7e9
lib/protocol: Fix OOR panic on recv-only folders (#7143) 2020-11-23 18:37:27 +01:00
Simon Frei
8ebd893349
lib/db: Store versions for last successful db migration (#7140) 2020-11-23 18:31:32 +01:00
Simon Frei
db1f20603a
lib/model: Retain index info for new folders/devs (ref #7100) (#7133) 2020-11-20 15:53:13 +01:00
Simon Frei
24af89c8e2
all: Refactor preparing configuration (#7127) 2020-11-20 14:21:54 +01:00
Simon Frei
641b7aee38
lib/db: Reset index-id when dropping device (#7135) 2020-11-20 14:17:09 +01:00
Simon Frei
b43eccf2fe
lib/model: Never send unpaused folder without index info (#7134) 2020-11-20 14:13:50 +01:00
Simon Frei
e11b309379
lib/model, lib/util: Replace IsComplete with NoRestartErr (ref #6947) (#7126) 2020-11-18 13:43:57 +01:00
Simon Frei
2d3a535ced
lib/model: Handle cluster-config before folder start (fixes #7122) (#7123) 2020-11-17 15:30:21 +01:00
Simon Frei
9524b51708
all: Implement suture v4-api (#6947) 2020-11-17 13:19:04 +01:00
Simon Frei
e8fc465ea8
lib/config: Use correct var in MaxConcurrentIncomingRequestKiB() (#7121) 2020-11-16 16:51:51 +01:00
Jakob Borg
cca73de6a1 lib/osutil: Consider sync() best effort (fixes #7117) 2020-11-14 09:23:27 +01:00
Simon Frei
b8051fb37e
lib/model: Prevent test deadlock (#7110) 2020-11-10 12:32:07 +01:00
Simon Frei
1d3b9876f6
lib/api, lib/db: Add file debug endpoint (#7095) 2020-11-10 09:24:45 +01:00
Rahmi Pruitt
2f6a25a56f
gui: Add advance config port mapping to gui (fixes #4824) (#7017) 2020-11-10 09:24:11 +01:00
Jakob Borg
846b265430
lib/tlsutil: Add O and OU to generated certificates (fixes #7108) (#7109) 2020-11-09 17:02:56 +01:00
Simon Frei
31559e908b
all: Add untrusted folders behind feature flag (ref #62) (#7055) 2020-11-09 15:33:32 +01:00
Simon Frei
54643e86b5
lib/model: Fix locking when resending cluster-configs (#7107) 2020-11-09 14:05:21 +01:00
Simon Frei
1f1729ba43
lib/model: Add done chan to track folder-lifetime (fixes #6664) (#7094) 2020-11-09 09:05:48 +01:00
Simon Frei
d4ce0dfd84
lib/model: Send indexes for newly shared folder (fixes #7098) (#7100) 2020-11-09 08:58:46 +01:00
Simon Frei
cc9ea9db89
lib/folder: Clear pull errors when nothing is needed anymore (#7093) 2020-11-06 14:22:20 +01:00
Simon Frei
a08a1b6998
lib/api: Fix debug endpoints (ref #7001) (#7092) 2020-11-06 14:21:37 +01:00
Simon Frei
d0ccea0404
lib/config: Sanity checks on MaxConcurrentWrites (ref #7064) (#7069) 2020-11-03 19:09:32 +01:00
Simon Frei
a38b370c8d
lib/ur: Fix panics in failure-reporting (fixes #7090) (#7091) 2020-11-03 12:29:33 +01:00
Jakob Borg
7892547873
lib: Remove USE_BADGER experiment (#7089)
This removes the switch for using a Badger database, because it has bugs
that it seems there is no interest in fixing, and no actual bug tracker
to track them in.

It retains the actual implementation for the sole purpose of being able
to do the conversion back to LevelDB if anyone is actually running with
USE_BADGER. At some point in a couple of versions we can remove the
implementation as well.
2020-11-03 09:10:35 +01:00
Simon Frei
4d1bcd718c
lib/api: Fix /rest/config path and add methods to cors (ref #7001) (#7081) 2020-11-01 21:36:54 +01:00
André Colomb
7dc0c6ab43
lib/api: Allow OPTIONS method in CORS preflight request handling (ref #7017) (#7079)
This allows for checking GUI / API availability without actually doing
a GET or POST request.
2020-11-01 14:29:55 +01:00
Tomasz Wilczyński
4a616f3cb2
lib/config: Check for "msdos" when detecting FAT FS in Android (#7072) 2020-10-30 15:13:56 +01:00
Simon Frei
9189c79d74
lib/api: Add missing config mod. locks (ref #7001) (#7053) 2020-10-23 10:34:20 +02:00
Simon Frei
f0f60ba2e7
lib/api: Add /rest/config endpoint (fixes #6540) (#7001) 2020-10-22 19:54:35 +02:00
Simon Frei
1c2be84e4e
lib/model: Pass device infos as struct (fixes #7051) (#7052) 2020-10-22 13:05:31 +02:00
Simon Frei
2ba3be5e4d
lib/db: Add mechanism to repair db without schema update (ref #7044) (#7047) 2020-10-21 14:21:09 +02:00
Simon Frei
5c91723ef2
lib/model: Handle index sender lifetime (fixes #7034) (#7038) 2020-10-21 11:51:53 +02:00
Audrius Butkevicius
a17a8cd48b
lib/connections: Fix LAN addresses begin advertised even when disabled (fixes #7035) (#7045) 2020-10-21 09:16:44 +02:00
Simon Frei
27c91c57d5
lib/db: Remove need for the right dev removing globals (fixes #7036) (#7044) 2020-10-21 08:26:10 +02:00
Simon Frei
1a8c10a8d0
lib/model: Use winning version instead of merge on conflict (#6995) 2020-10-19 08:53:19 +02:00
Simon Frei
01a7ef3b0f
lib/db: Undo adding user info to panic msgs (ref #7029) (#7040) 2020-10-19 08:40:37 +02:00
Simon Frei
23c935b05a
lib/db: Ignore not found on delete in recalcGlobal (ref #7026) (#7041) 2020-10-19 08:28:53 +02:00
Simon Frei
d828adb648
cmd/stcrashreceiver, lib/db: Improve panic message handling (#7029) 2020-10-08 17:37:45 +02:00
Simon Frei
9d09fd6af3
all: Add failure reporting (fixes #7028) (#6925) 2020-10-07 10:05:13 +02:00
Simon Frei
14ae330eff
lib/db: Improve error handling checking db (ref #7026) (#7027) 2020-10-06 20:14:09 +02:00
Simon Frei
42de53c6c9
lib/model: Fix race in GlobalDirectory tests (fixes #7021) (#7022) 2020-10-03 20:46:17 +02:00
Simon Frei
48da6f0f22
lib: Use uint64 for disk stats (ref #3930) (#7019) 2020-10-02 15:22:28 +02:00
Simon Frei
a20c6ca868
lib/model, lib/protocol: Send ClusterConfig on config change (fixes #7020) (#7018) 2020-10-02 11:49:51 +02:00
Audrius Butkevicius
e027175446
all: Move remaining protos to use the vanity plugin (#7009) 2020-10-02 08:07:05 +02:00
Jakob Borg
7774932302
lib/config, lib/syncthing: Only drop delta index on upgrade if so ordered (fixes #6982) (#6983) 2020-09-30 20:16:30 +02:00
Simon Frei
1b1d38183d
lib: Remove HelloResult type, same as Hello (#7015) 2020-09-29 12:17:38 +01:00
Tomasz Wilczyński
fb3281b647
lib/fs: Add COM0 and LPT0 to invalid Windows file names (ref #7008) (#7013)
COM0 and LPT0 are not listed in the official Microsoft's documentation
at https://docs.microsoft.com/windows/win32/fileio/naming-a-file, but
in reality are also invalid in Windows Explorer.

Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
2020-09-28 20:35:54 +02:00
Jakob Borg
e6b1f67ecf
lib/fs: Be even more strict about Windows names (ref #7008) (#7012)
Things like nul.whatever.txt are also not allowed.
2020-09-28 10:42:37 +02:00
Jakob Borg
9e0b924d57
lib/fs: Be more clear about invalid file names (ref #7010) (#7011)
Add specific errors for the failures, resulting in this rather than just
the generic "invalid filename":

[MRIW7] 08:50:50 INFO: Puller (folder default, item "NUL"): syncing: filename is invalid: name is reserved
[MRIW7] 08:50:50 INFO: Puller (folder default, item "fail."): syncing: filename is invalid: name ends with space or period
[MRIW7] 08:50:50 INFO: Puller (folder default, item "sup:yo"): syncing: filename is invalid: name contains reserved character
[MRIW7] 08:50:50 INFO: default: Failed to sync 3 items
2020-09-28 10:22:50 +02:00
Jakob Borg
df99237a7f
lib/model: Disallow reserved names, and names ending with period on Windows (fixes #7008) (#7010)
Like we already do with names ending in space or containing invalid
characters.
2020-09-28 08:25:43 +01:00
Simon Frei
8452fd2ab4
lib/model, lib/scanner: Prevent races aborting scans (fixes #6994) (#6997) 2020-09-25 11:27:44 +02:00
Tobias Klauser
c390565eef
lib/fs: Use file clone ioctl wrappers and types from golang.org/x/sys/unix (#7000)
Use the IoctlFileClone and IoctlFileCloneRange ioctl wrappers and the
FileCloneRange type provided by golang.org/x/sys/unix instead of
locally implementing them. This also allows to re-enable the code for
ppc/ppc64/ppc64le again (see commit 758a1a6a37 ("lib/fs: Disable ioctl
on ppc (fixes #6898) (#6901)")) since golang.org/x/sys/unix internally
uses the correct FICLONE and FICLONERANGE values depending on $GOARCH.
2020-09-24 10:29:32 +02:00
Simon Frei
8210466b03
lib/model: Consider case conflicts when checking to be deleted items (#6986) 2020-09-12 07:45:50 +02:00
greatroar
3e24d82513
lib/fs: Optimize UnicodeLowercase (#6979)
Most notably, it now detects all-lowercase files and returns these
as-is. The tests have been expanded with two cases and are now used
as a benchmark (admittedly a rather trivial one).

name                           old time/op    new time/op    delta
UnicodeLowercaseMaybeChange-8    4.59µs ± 2%    4.57µs ± 1%    ~     (p=0.197 n=10+10)
UnicodeLowercaseNoChange-8       3.26µs ± 1%    3.09µs ± 1%  -5.27%  (p=0.000 n=9+10)
2020-09-11 09:16:10 +02:00
Simon Frei
08bebbe59b
lib/db, lib/syncthing: Don't repair DB on upgrade, but on error (fixes #6917) (#6971) 2020-09-10 10:54:41 +02:00
Simon Frei
c5c23ed10f
lib/model: Consider existing file when handling symlink on windows (#6977) 2020-09-10 10:52:38 +02:00
Simon Frei
286698ccb1
lib/model: Handle symlink deletion on windows (fixes #6972) (#6976) 2020-09-10 08:25:44 +02:00
Simon Frei
56d48d341f
lib/model: Fix case-only renames on pull (#6978) 2020-09-10 08:23:54 +02:00
Jakob Borg
780fb3bac1
lib/fs: More efficient casefs cache (#6974)
This changes the cache to cache less things, yet retain the required
efficiency for our walk usecase. This uses less memory.

Specifically, instead of keeping result and child caches for each path
level, only keep a single cached child. In practice our operations are
depth-first, or almost depth-first, and then we retain the same hit
ratio for a smaller cache size.

I improved the benchmark so that it counts the Lstat and DirNames
operations performed, and they do not change significantly. The amount
of allocated memory is reduced by 20% and the walk itself is actually
slightly faster.

This also removes the clear based on number of cached names (as that is
not a thing any more) and the timer based clear (which was unused). This
means we'll retain the last cache state forever until it's cleared by a
write operation, but we did that before too and that state is now a lot
smaller...

The overhead compared to not using a casefs, for our typical "double
walk" (walk the tree then stat everything again) is 2x the dirnames we
would otherwise call, and no overhead on the stats (unchanged from old
implementation)

```
name                         old time/op         new time/op         delta
WalkCaseFakeFS100k/rawfs-8           306ms ± 1%          305ms ± 2%     ~     (p=0.182 n=9+10)
WalkCaseFakeFS100k/casefs-8          579ms ± 5%          557ms ± 1%   -3.77%  (p=0.000 n=10+10)

name                         old B/entry         new B/entry         delta
WalkCaseFakeFS100k/rawfs-8             590 ± 0%            590 ± 0%     ~     (all equal)
WalkCaseFakeFS100k/casefs-8          1.09k ± 0%          0.87k ± 0%  -19.98%  (p=0.000 n=10+10)

name                         old DirNames/entry  new DirNames/entry  delta
WalkCaseFakeFS100k/rawfs-8            0.51 ± 0%           0.51 ± 0%     ~     (all equal)
WalkCaseFakeFS100k/casefs-8           1.02 ± 0%           1.02 ± 0%     ~     (all equal)

name                         old DirNames/op     new DirNames/op     delta
WalkCaseFakeFS100k/rawfs-8           51.2k ± 0%          51.2k ± 0%     ~     (all equal)
WalkCaseFakeFS100k/casefs-8           102k ± 0%           102k ± 0%     ~     (all equal)

name                         old Lstat/entry     new Lstat/entry     delta
WalkCaseFakeFS100k/rawfs-8            3.02 ± 0%           3.02 ± 0%     ~     (all equal)
WalkCaseFakeFS100k/casefs-8           3.02 ± 0%           3.02 ± 0%     ~     (all equal)

name                         old Lstat/op        new Lstat/op        delta
WalkCaseFakeFS100k/rawfs-8            302k ± 0%           302k ± 0%     ~     (all equal)
WalkCaseFakeFS100k/casefs-8           302k ± 0%           302k ± 0%     ~     (all equal)

name                         old allocs/entry    new allocs/entry    delta
WalkCaseFakeFS100k/rawfs-8            15.7 ± 0%           15.7 ± 0%     ~     (all equal)
WalkCaseFakeFS100k/casefs-8           27.5 ± 0%           26.1 ± 0%   -5.09%  (p=0.000 n=10+10)

name                         old ns/entry        new ns/entry        delta
WalkCaseFakeFS100k/rawfs-8           2.02k ± 1%          2.02k ± 2%     ~     (p=0.163 n=9+10)
WalkCaseFakeFS100k/casefs-8          3.83k ± 5%          3.68k ± 1%   -3.77%  (p=0.000 n=10+10)

name                         old alloc/op        new alloc/op        delta
WalkCaseFakeFS100k/rawfs-8          89.2MB ± 0%         89.2MB ± 0%     ~     (p=0.364 n=9+10)
WalkCaseFakeFS100k/casefs-8          164MB ± 0%          131MB ± 0%  -19.97%  (p=0.000 n=10+10)

name                         old allocs/op       new allocs/op       delta
WalkCaseFakeFS100k/rawfs-8           2.38M ± 0%          2.38M ± 0%     ~     (all equal)
WalkCaseFakeFS100k/casefs-8          4.16M ± 0%          3.95M ± 0%   -5.05%  (p=0.000 n=10+10)
```
2020-09-09 14:38:39 +02:00
Simon Frei
e3cd9219b8
lib/model: Don't fail over case-conflict on tempfile (fixes #6973) (#6975) 2020-09-09 11:47:14 +02:00
Simon Frei
b628ec5054
lib/fs: Use generic case resolver on windows (fixes #6968) (#6969) 2020-09-08 13:10:26 +02:00
Simon Frei
ecc24428ac
lib/model: Prevent duplicate need count on recalculating metadata (#6964) 2020-09-07 20:19:03 +02:00
Jakob Borg
674fca3868
lib/db, lib/protocol: Never need empty-version entries (fixes #6961) (#6962)
Avoid havoc when discovering locally-deleted-upgraded files during repair / need calculation...

Co-authored-by: Simon Frei <freisim93@gmail.com>
2020-09-07 20:18:25 +02:00
Simon Frei
e19728d8cc
lib/model: Correct completion perc. when globalBytes is 0 (#6963) 2020-09-07 20:03:18 +02:00
Jakob Borg
415adfbae6
lib/db: Don't hang on double iterator release with Badger (#6960)
Since iterators must be released before committing or discarding a
transaction we have the pattern of both deferring a release plus doing
it manually. But we can't release twice because we track this with a
WaitGroup that will panic when there are more Done()s than Add()s. This
just adds a boolean to let an iterator keep track.
2020-09-07 10:31:33 +02:00
Simon Frei
3dd13c3994
test, lib/model: Various integration test updates & improvements (#6956) 2020-09-07 09:35:37 +02:00
Simon Frei
52d1681fe6
lib/db: Copy returned, old FileVersion (#6952) 2020-09-04 14:13:38 +02:00
Simon Frei
7b821d2550
lib/db: Add local need check&repair (#6950) 2020-09-04 14:01:46 +02:00
Simon Frei
0e3e0a7c9e
cmd/stindex, lib/db: Use same condition to check for needed files (#6954) 2020-09-04 13:59:04 +02:00
Jakob Borg
71bfad0bc6 cmd/stindex: Add dumping folder meta 2020-09-04 11:39:21 +02:00
Jakob Borg
b7986801cd cmd/stindex: Print more hashes and versions 2020-09-04 09:10:33 +02:00
Simon Frei
6365e6108f
lib/model: Don't acquire I/O token for send-only pull (#6951) 2020-09-03 13:59:45 +02:00
Simon Frei
540518a7b7
lib/model: Add missing lock on download-state (fixes #6880) (#6945) 2020-08-30 08:03:10 +02:00
Jakob Borg
3ffbe45a6d
lib/db, lib/model: Avoid accounting mishap on folder restart (fixes #6938) (#6939)
We created a new fileset before stopping the folder during restart. When
we create that fileset it loads the current metadata and sequence
numbers from the database. But the folder may have time to update those
before stopping, leaving the new fileset with bad data.

This would cause wrong accounting (forgotten files) and potentially
sequence reuse causing files not sent to other devices.

This change reuses the fileset on restart, skipping the issue entirely.
It also moves the creation of the fileset back under the lock so there
should be no chance of concurrency issues here.
2020-08-27 18:33:27 +02:00
Jakob Borg
3393db1f69
lib/model: Save config after auto accepting folders (fixes #6922) (#6937) 2020-08-27 16:28:19 +02:00
Jakob Borg
145d87ce70 lib/model: Fixup spelling of protocol.FileIntf in test (ref #6463) 2020-08-27 16:21:04 +02:00
Simon Frei
06ac631351
lib/model: Reduce fmut locking in ClusterConfig (#6913)
The FileSet.Drop operation in there needs to potentially update a whole lot of global lists, which can take a while (longer than the deadlock interval apparently)
2020-08-27 16:01:46 +02:00
Simon Frei
1fc2dbdeeb
lib/model: Add test for recv-only changes undone by remote (#6463)
Co-authored-by: Audrius Butkevicius <audrius.butkevicius@gmail.com>
2020-08-27 15:54:00 +02:00
Simon Frei
3f0eba388c
lib/syncthing: Also cleanup on startup error (#6926) 2020-08-27 15:52:51 +02:00
Audrius Butkevicius
b19b5c95d3
lib/connections: Announce LAN addresses by default (fixes #6928) (#6896) 2020-08-25 11:48:14 +02:00
Audrius Butkevicius
d507d932b8
all: Use protobuf to generate config structs (fixes #6734) (#6900) 2020-08-25 08:11:14 +02:00
Rahmi Pruitt
5b953033c7
Created cleanup functionality for syncthing (#6884)
* Add clean up for Simple File Versioning pt.1

created test

* Add clean up for Simple File Versioning pt.2

Passing the test

* stuck on how javascript communicates with backend

* Add trash clean up for Simple File Versioning

Add trash clean up functionality of to allow the user to delete backups
after specified amount of days.

* Fixed html and js style

* Refactored cleanup test cases

Refactored cleanup test cases to one file and deleted duplicated code.

* Added copyright to test file

* Refactor folder cleanout to utility function

* change utility function to package private

* refactored utility function; fixed build errors

* Updated copyright year.

* refactor test and logging

* refactor html and js

* revert style change in html

* reverted changes in html and some js

* checkout origin head version edit...html

* checkout upstream master and correct file
2020-08-24 12:14:30 +01:00
Simon Frei
a65b46debd
lib/model: Don't mark globally deleted items as recv-only (fixes #6910) (#6911) 2020-08-21 12:34:22 +02:00
Simon Frei
b628460930
lib/model: Prevent localflag flipflopping on del. item (ref #6865) (#6919) 2020-08-21 10:09:48 +02:00
Simon Frei
cc1f6e4d4a
lib/db, lib/model: Cover exec-paths with debug logging (#6918) 2020-08-20 16:11:20 +02:00
Simon Frei
88599bc154
lib/model: Handle del. dir with locally changed items on pull (fixes #6873) (#6914) 2020-08-20 10:56:29 +02:00
Simon Frei
fc2c46e82f
lib/config, lib/fs: Make junction behaviour configurable (ref #6606) (#6907) 2020-08-19 19:58:51 +02:00
Simon Frei
ce4d149bf5
lib/nat: Don't hang on draining timer chan (fixes #6908) (#6912) 2020-08-19 15:58:44 +01:00
Audrius Butkevicius
cbbc262161
lib/db: Dont recurse on flush (fixes #6905) (#6906)
Or else we crash.

Co-authored-by: Jakob Borg <jakob@kastelo.net>
Co-authored-by: Simon Frei <freisim93@gmail.com>
2020-08-19 13:13:44 +02:00
Audrius Butkevicius
758a1a6a37
lib/fs: Disable ioctl on ppc (fixes #6898) (#6901) 2020-08-19 07:56:35 +02:00
Audrius Butkevicius
d0c6c18b4f
lib/dialer: Try dialing without reuse in parallel (fixes #6892) (#6893) 2020-08-18 13:11:12 +02:00
Audrius Butkevicius
bf9ff17267
all: Remove need to restart syncthing (#6883) 2020-08-18 09:26:33 +02:00
Simon Frei
8f5215878b
lib/db: Don't put truncated files (ref #6855, ref #6501) (#6888) 2020-08-18 09:20:12 +02:00
Simon Frei
9c2117f08e
lib/osutil: Check returned error instead of info (ref #6885) (#6887) 2020-08-12 11:33:56 +02:00
Simon Frei
fd8bea6179
lib/osutil: Preserve perms in AtomicWriter (fixes #tbd) (#6885) 2020-08-10 18:48:37 +02:00
Audrius Butkevicius
e9bb17307d
lib/fs: Unwrap mtimeFile, get fd the "correct" way (ref #6875) (#6877) 2020-08-07 07:47:48 +02:00
Jakob Borg
5e1f39b6f6 lib/fs: Fix WatchRename test for FreeBSD (fixes #6613) 2020-08-03 23:24:01 +02:00
Audrius Butkevicius
96e197e502
lib/model: Don't close file early (fixes #6875) (#6876) 2020-08-03 21:54:42 +02:00
Simon Frei
bbf25e2d02
lib/db: Log context on panic (#6872) 2020-08-01 17:32:36 +02:00
Simon Frei
a85bc1c3a6
lib/model: Check folder error before sync-waiting (fixes #6793) (#6847) 2020-07-31 19:26:09 +02:00
Simon Frei
850dd4cd25
lib/db: Don't count invalid items to global state (ref #6850) (#6863) 2020-07-30 13:49:14 +02:00
Simon Frei
e46c8ab9ee
lib/model: Detect deleted RO items scanning on non-RO (fixes #6864) (#6865) 2020-07-30 13:41:45 +02:00
Jakob Borg
2dc2aa5d21
lib/connections, lib/tlsutil: Handle certName in Go 1.15 (fixes #6867) (#6868)
Our authentication is based on device ID (certificate fingerprint) but
we also check the certificate name for ... historical extra security
reasons. (I don't think this adds anything but it is what it is.) Since
that check breaks in Go 1.15 this change does two things:

- Adds a manual check for the peer certificate CommonName, and if they
  are equal we are happy and don't call the more advanced
  VerifyHostname() function. This allows our old style certificates to
  still pass the check.

- Adds the cert name "syncthing" as a DNS SAN when generating the
  certificate. This is the correct way nowadays and makes VerifyHostname()
  happy in Go 1.15 as well, even without the above patch.
2020-07-30 13:36:11 +02:00
Jakob Borg
d53a2567a4
build: Actually set build tags (#6866)
Apparently our Tags field depended on having specific files react to
tags and add themselves there. This, instead, works for all tags.

Also, pass tags to the test command line.
2020-07-30 10:58:43 +02:00
Jakob Borg
6f4671ed27 lib/connections: Add noquic tag
The QUIC package is notorious for being incompatible with either too
old or too new Go releases. Currently it doesn't build with Go 1.15 RC
and I want to test the rest with Go 1.15. With this I can do `go run
build.go --tags noquic` to do that.
2020-07-30 09:42:02 +02:00
Simon Frei
424d1b1608
lib/db: Commit meta when dropping device (#6862) 2020-07-28 16:46:42 +02:00
Simon Frei
1b9e5c0937
lib/db: Include blocks in db check (ref #6855) (#6861) 2020-07-28 16:25:07 +02:00
Jakob Borg
62cff26edf lib/config: Repair versioning XML serialization (fixes #6859, ref #6693) 2020-07-28 12:08:37 +02:00
Simon Frei
932d8c69de lib/fs: Properly handle case insensitive systems (fixes #1787, fixes #2739, fixes #5708)
With this change we emulate a case sensitive filesystem on top of
insensitive filesystems. This means we correctly pick up case-only renames
and throw a case conflict error when there would be multiple files differing
only in case.

This safety check has a small performance hit (about 20% more filesystem
operations when scanning for changes). The new advanced folder option
`caseSensitiveFS` can be used to disable the safety checks, retaining the
previous behavior on systems known to be fully case sensitive.

Co-authored-by: Jakob Borg <jakob@kastelo.net>
2020-07-28 11:15:11 +02:00
Simon Frei
08e0f938a9
lib/db: Update global meta even if unchanged (fixes #6850) (#6852) 2020-07-24 12:36:16 +02:00
Simon Frei
ebead944b5
lib/fs: Pass infinite recursion error on instead of warning (#6846)
Prompted by https://forum.syncthing.net/t/infinite-filesystem-recursion-detected/15285. In my opinion the filesystem shouldn't throw warnings but pass on errors for the caller to decide what's to be happening with it. Right now in this PR an infinite recursion is a normal scan error, i.e. folder is in failed state and displays failed items, but no warning. I think that's appropriate but if deemed appropriate an additional warning can be thrown in the scanner.
2020-07-22 22:10:24 +02:00
Audrius Butkevicius
55147f5901
lib/db: Rework flush hooks (#6838) 2020-07-19 08:55:27 +02:00
Jakob Borg
aedc2d788f
lib/model, lib/versioner: Drive version cleanup from scanner (fixes #6313) (#6693)
This change adds a separate config for the cleanup interval, and runs that cleanup from the main folder loop.
2020-07-14 10:48:50 +02:00
Simon Frei
16f4921c50
lib/nat: Make sure nat keeps being disabled (fixes #6823) (#6824) 2020-07-14 08:16:08 +02:00
Jakob Borg
fc1dac5196
lib/scanner: Less strict validation (fixes #6827) (#6828)
This fixes the change in #6674 where the weak hash became a deciding
factor. Now we again just use it to accept a block, but don't take a
negative as meaning the block is bad.
2020-07-11 14:48:45 +02:00
greatroar
9f92f8c609
lib/db: Use SipHash to deal with hash collision in GC (#6826)
If the GC finds a key k that it wants to keep, it records that in a
Bloom filter. If a key k' can be removed but its hash collides with k,
it will be kept. Since the old Bloom filter code was completely
deterministic, the next run would encounter the same collision, assuming
k must still be kept.

A randomized hash function that uses all the SHA-256 bits solves this
problem: the second run has a non-zero probability of removing k', as
long as the Bloom filter is not completely full.
2020-07-11 09:36:09 +02:00
Jakob Borg
bbda58a29f lib/db: Add test for GC run 2020-07-11 09:22:15 +02:00
Simon Frei
8d67235a75
lib/config: Don't cache device map (fixes #6816) (#6817) 2020-07-07 23:44:49 +02:00
Jakob Borg
0c61c66511
lib/api, lib/model: Improve folder completion API (fixes #6075) (#6808) 2020-07-03 08:48:37 +02:00
Simon Frei
ebbe1abe28
lib/protocol: Include WeakHash in BlocksHash (#6799) 2020-06-29 09:33:07 +02:00
Audrius Butkevicius
9d4a700829
lib/ur: Track new folder feature usage (#6803) 2020-06-28 20:35:22 +02:00
Simon Frei
d7fc7008af
lib/protocol: Add BlocksHash to FileInfo string (#6802) 2020-06-27 07:44:33 +02:00
Simon Frei
9c45ac381c
lib/model: Consider weak hash on recheckFile (#6797) 2020-06-26 16:47:03 +02:00
Simon Frei
4f06708330
lib/api, lib/model: Fixes around event request tracking (#6774) 2020-06-25 21:48:48 +02:00
Simon Frei
80ada1bb6c
lib/protocol: Don't warn when dropping deleted invalid item (fixes #6795) (#6796) 2020-06-25 20:24:31 +02:00
Simon Frei
0648fb0626
lib/model: Don't force rescan already changed items (#6798) 2020-06-25 20:23:59 +02:00
Simon Frei
90e248615f
lib/scanner: Test weak hash consistency (ref #5556) (#6794)
Relevant much earlier changes:
  9b1c592fb7
  bd1c29ee32

Make sure vanilla and rolling adler are consistent. And that they match
with scanner.Validate.
2020-06-25 14:47:35 +02:00
Audrius Butkevicius
bb76311ec6
lib/ur: Zero our the value before unmarshal (#6790)
* lib/ur: Zero our the value before unmarshal

* Comment

* Complete rewrite
2020-06-23 21:23:32 +01:00
Audrius Butkevicius
689cf2a5ee
lib/ur: Normalise contract between syncthing and ursrv (#6770)
* Fix ui, hide report date

* Undo Goland madness

* UR now web scale

* Fix migration

* Fix marshaling, force tick on start

* Fix tests

* Darwin build

* Split "all" build target, add package name as a tag

* Remove pq and sql dep from syncthing, split build targets

* Empty line

* Revert "Empty line"

This reverts commit f74af2b067dadda8a343714123512bd545a643c3.

* Revert "Remove pq and sql dep from syncthing, split build targets"

This reverts commit 8fc295ad007c5bb7886c557f492dacf51be307ad.

* Revert "Split "all" build target, add package name as a tag"

This reverts commit f4dc88995106d2b06042f30bea781a0feb08e55f.

* Normalise contract types

* Fix build add more logging
2020-06-23 09:47:15 +01:00
Simon Mwepu
72f954dcab
lib/versioner: Create versioning directory recursively (fixes #6565) (#6678)
Co-authored-by: Jakob Borg <jakob@kastelo.net>
2020-06-23 08:37:29 +02:00
Simon Frei
85794933d3
lib/model: Remove double error handling in performFinish (#6780) 2020-06-23 07:30:16 +02:00
Simon Frei
4881a6336f
lib/db: Correct need accounting on reset (fixes #6784) (#6785) 2020-06-23 07:29:27 +02:00
Audrius Butkevicius
5fb3992275
lib/model: More connection closing (#6778)
* lib/model: More connection closing

* Revert "lib/model: More connection closing"

This reverts commit 5397c3a55c0de25fab93d4ec622530818f1b2a46.

* Add tests, fix one broken-ness

* Update lib/model/model.go

Co-authored-by: Simon Frei <freisim93@gmail.com>

* Update model.go

* Update model.go

Co-authored-by: Simon Frei <freisim93@gmail.com>
2020-06-22 21:26:26 +01:00
Jakob Borg
b62b7d269e
lib/connections: Correctly fixup port in IPv6 listen URLs (#6786)
Previously tcp://[fe80::1] would get "fixed" to tcp://[[fe80::1]]:22000
which is not great.
2020-06-22 16:47:15 +01:00
Audrius Butkevicius
cbaef624cf
lib/nat: Make service termination faster (#6777)
* lib/nat: Make service termination faster

* Newline
2020-06-22 09:01:57 +01:00
greatroar
dc145bfad7
lib/protocol: Use sha256.Sum256 in NewDeviceID (#6775)
This is shorter, skips two allocations, makes the function inlineable
and is safer, since the compiler now check whether
DeviceIDLength == sha256.Size.
2020-06-21 20:38:06 +02:00
greatroar
d985aa9e4b
lib/scanner: Fix Validate docs (#6776)
Co-authored-by: greatroar <@>
2020-06-21 19:23:06 +01:00
Jakob Borg
705710b1a1
gui, lib/api: Add more version info to about dialog (#6773) 2020-06-21 12:20:19 +02:00
Simon Frei
f66e57947b
lib/model: Use right db snap when scanning recvonly folder (#6769) 2020-06-21 09:28:29 +02:00
Simon Frei
2716898cb9
lib/model: Don't ignore stat failure in performFinish (#6766) 2020-06-19 23:47:29 +02:00
Audrius Butkevicius
deaccc7f8d
lib/fs: Add support for Windows duplicate extents (#6764) 2020-06-18 22:32:26 +01:00
Simon Frei
22f0077262
lib/model: Don't stay scanning forever on fail (#6761) 2020-06-18 14:13:46 +01:00