Commit Graph

1840 Commits

Author SHA1 Message Date
greatroar
3746c899b7
build: List go:generate tools in tools.go (#7599) 2021-05-08 12:52:06 +02:00
Audrius Butkevicius
87a0eecc31
lib/fs, lib/api, lib/model: Expose mtime remappings as part of /db/file (#7624)
* lib/fs, lib/api, lib/model: Expose mtime remappings as part of /db/file

* Fix wrong error returned by CLI

* Gofmt

* Better names

* Review comments

* Review comments
2021-05-03 11:28:25 +01:00
Simon Frei
75b58eb480
lib/fs: Watch attrib. changes on inotify for mod. time (#7623)
* lib/fs: Watch attrib. changes on inotify for mod. time

* fix bsds (real) and darwin (test only)
2021-05-02 16:45:44 +02:00
Simon Frei
dd39556759
lib: Revert needing invalid files (fixes #7608, ref #7476) (#7609) 2021-04-29 22:01:46 +02:00
Simon Frei
1f30383866
lib/model: Remove path from enc errors and report only once (#7610) 2021-04-29 19:21:07 +02:00
Gahl Saraf
66662cd678
Trigger connection loop on config device addition (fixes #7600) (#7604)
* Trigger connection loop on config device addition (fixes #7600)

* Also check for device address equality

* Move EqualStrings from api_test to utils, and use in connections/service.go

* Make sure CommitConfiguration cannot block due on the deviceAddressesChanged channel

* Update lib/connections/service.go

Co-authored-by: Jakob Borg <jakob@kastelo.net>
2021-04-26 21:13:59 +01:00
Simon Frei
8734fa65fc
lib/model: Handle invalid needed items on send-only (ref #7476) (#7596) 2021-04-26 15:36:51 +02:00
Jakob Borg
74823e81e9
all: Deprecate TLS 1.2 on sync connections (fixes #7594) (#7598)
This makes us use TLS 1.3+ on sync connections by default. A new option
`insecureAllowOldTLSVersions` exists to allow communication with TLS
1.2-only clients (roughly Syncthing 1.2.2 and older). Even with that
option set you get a slightly simplified setup, with the cipher suite
order fixed instead of auto detected.
2021-04-26 10:04:35 +02:00
Jakob Borg
e4c995a321
lib/model: Don't remove non-empty .stfolder (#7578) 2021-04-15 18:13:35 +02:00
André Colomb
130d14cec9
api: Log API authorization failures. (#7575) 2021-04-15 07:33:02 +02:00
Simon Frei
15d2dc3a4f
lib/connections: Add SyscallConn() to quic conn (fixes #7551) (#7570) 2021-04-13 12:59:58 +01:00
André Colomb
f6df1a760d
lib/api: Log the remote address on login attempts (#7560)
This enables usage of the audit log to e.g. automatically block remote
addresses from connecting after repeated login failures.
2021-04-13 10:14:44 +02:00
Simon Frei
f71fcd440a
all: Remove untrusted feature flag (fixes #109) (#7567)
No longer hide the web UI controls for the new untrusted/encrypted
device feature. Testing hasn't been very widespread, but there has been
some and quite a few bugs have been caught and fixed. I believe its time
to not hide it anymore, and cautiously recommend usage. E.g. mention
that the feature hasn't been widely used yet and anyone using it is an
early adopter, but drop the bit about not using it with production data.
We can maybe stress the need for backups in general and especially
using this.
2021-04-13 10:12:56 +02:00
Simon Frei
1a00ea7c6e
lib: Prevent using protocol method with native path (fixes #7557) (#7563) 2021-04-11 15:29:43 +02:00
Simon Frei
ec0a66c75b
lib/db, lib/model: Refactor removing expired pending folders (#7537) 2021-04-11 15:24:08 +02:00
Jakob Borg
8e38ecdeb2
Merge pull request from GHSA-x462-89pf-6r5h 2021-04-06 08:00:00 +02:00
Simon Frei
f30f9c50f8
lib/db: Handle indirection error repairing sequences (fixes #7026) (#7525) 2021-04-05 10:24:16 +02:00
Simon Frei
0dcd9794d4
lib/protocol: Deterministic encrypted version (fixes #7533) (#7538) 2021-03-31 08:59:15 +02:00
André Colomb
fb2d85b9d5
lib/db: Fix comparison of pending folder timestamps (fixes #7532) (#7535) 2021-03-30 21:24:53 +02:00
Simon Frei
7f0d4f6ba8
lib/db: Don't panic debugging an inexistent file (#7534) 2021-03-30 20:06:01 +02:00
Simon Frei
0d7a77ba85
lib/model: Reset conn when enc token is missing (fixes #7198) (#7518) 2021-03-23 10:38:40 +01:00
Simon Frei
924b96856f
lib: Handle adding enc folders on an existing conn (fixes #7509) (#7510) 2021-03-22 21:50:19 +01:00
Simon Frei
6b25eb2e79
lib/ur: Prevent panic when blocksResult is nil (ref #7495) (#7515) 2021-03-22 15:07:41 +01:00
Simon Frei
bc08a951f1
lib/model: Encrypted fileinfo trailer needs to be in wire format (#7505) 2021-03-21 10:34:08 +01:00
Lars Lehtonen
a87c5515bd
lib/model: Error check in test (#7508) 2021-03-21 10:32:17 +01:00
Jakob Borg
ebcd22b02b
lib/osutil: Fix raising max FDs on macOS (#7507)
There was a logic mistake, so the limit in question wasn't used. On my
macOS this doesn't seem to matter, the hard limit returned is 2^63-1 and
setting the soft limit to that works. However I'm assuming that's not
the case for older macOSes since it was so nicely documented, so we
should still have this working. (10240 FDs should be enough for
anybody.)
2021-03-20 16:32:36 +01:00
Audrius Butkevicius
4b02b7e6f1
lib/connections: Silence "connected to myself" messages. (#7500) 2021-03-17 23:53:20 +01:00
Jakob Borg
fdd823d2cb lib/osutil: Remove unused code 2021-03-17 23:18:07 +01:00
Jakob Borg
8ef504f745
all: Simplify some method calls (#7499)
strings.Replace(a, b, c, -1) -> strings.ReplaceAll(a, b, c)

(Go 1.12) and who knows what was up with that dialQueue.Sort() thing.
2021-03-17 23:12:26 +01:00
deepsource-autofix[bot]
6c573a5762
Remove unnecessary guard around delete (#7496)
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>
2021-03-17 22:23:36 +01:00
Jakob Borg
3ac858b150
all: Remove miscellaneous vestigial code (#7495) 2021-03-17 22:23:12 +01:00
Jakob Borg
f4372710bf
all: Remove crypto/md5 (#7493)
This is a mostly pointless change to make security scanners and static
analysis tools happy, as they all hate seeing md5. None of our md5 uses
were security relevant, but still. Only visible effect of this change is
that our temp file names for very long file names become slightly longer
than they were previously...
2021-03-17 22:22:49 +01:00
Jakob Borg
f39477bbd5
lib/api: Missing error handling in API delete-device (#7494) 2021-03-17 22:08:44 +01:00
Jakob Borg
6e5514419d
lib/db: Fix some omitted error checks, unused variable (#7489) 2021-03-17 21:41:07 +01:00
Jakob Borg
81484699db
lib/model: Actually break puller loop on context cancel (#7492)
Current break does nothing (breaks the select).
2021-03-17 21:34:52 +01:00
deepsource-autofix[bot]
0930bccf88
cmd/ursrv, lib/scanner: Remove unnecessary slicing of slices (#7491)
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>
2021-03-17 21:04:36 +01:00
Jakob Borg
e321bd3941
lib/*/auto: Add noassets files (#7490)
This adds a couple of dummy asset files protected by the "noassets"
build tag. The purpose is that it should be possible for, for example,
CI tools and static analysis things to compile and analyze the source
tree without our custom asset generation step. Also makes `go test -tags
noassets ./...` work without building assets first.
2021-03-17 21:03:35 +01:00
Simon Frei
4b02937862
lib/model: Add missing lock on availability func (#7487) 2021-03-17 20:46:13 +01:00
Simon Frei
3f2b584c4e
lib/model: Don't use ignore patterns for recv-enc folders (fixes #7469) (#7472) 2021-03-16 15:04:11 +01:00
Simon Frei
d2d4fcc1df
lib/protocol: Improve messages when an error occurs receiving (ref #7466) (#7470) 2021-03-15 19:14:09 +01:00
Simon Frei
273ee09925
lib/db, lib/model: Allow needing invalid files (fixes #7474) (#7476) 2021-03-15 07:58:01 +01:00
Simon Frei
f80ee472c2
lib/protocol: Set invalid flag on encrypted file infos (fixes #7466) (#7467) 2021-03-13 16:57:36 +01:00
Simon Frei
97a8777d03
lib/fs: Check both old and new path when renaming (fixes #7426) (#7463) 2021-03-12 21:15:50 +01:00
Simon Frei
8a4c00d82e
lib/model: Send failure report on CC encryption check error (#7460) 2021-03-12 12:21:54 +01:00
Simon Frei
31f859e909
lib/model: Return correct error in puller-iteration (ref #7424) (#7461) 2021-03-12 12:21:28 +01:00
Jakob Borg
4d979a1ce9
all: Truncate some timestamps (fixes #7457) (#7459)
This truncates times meant for API consumption to second precision,
where fractions won't typically matter or add any value. Exception to
this is timestamps on logs and events, and of course I'm not touching
things like file metadata.

I'm not 100% certain this is an exhaustive change, but it's the things I
found by grepping and following the breadcrumbs from lib/api...

I also considered general-but-ugly solutions, like having the API
serializer itself do reflection magic or even regexps on returned
objects, but decided against it because aurgh...
2021-03-12 10:35:10 +01:00
Simon Frei
4465cdf8bc
lib/api: Fix body of renamed config/restart-required endpoint (ref #7402) (#7453) 2021-03-11 15:54:05 +01:00
Simon Frei
3938b61c3f
lib/fs: Expose fs option on interface (fixes #7385, ref #7381) (#7389) 2021-03-11 15:23:56 +01:00
Jakob Borg
cdef503db6
all: Make config.Wrapper an actual suture.Service (fixes #7451) (#7452) 2021-03-11 14:51:00 +01:00
Jakob Borg
df08984a58
lib/api: Sanitize names used in certificates (fixes #7434) (#7435) 2021-03-11 13:15:03 +01:00
Simon Frei
9a001051d6
cmd/ursrv, lib/ur: Collect and present encryption usage (#7448) 2021-03-10 22:26:56 +00:00
greatroar
c00520281b
lib/protocol: Optimize FileKey (#7440) 2021-03-07 18:44:21 +01:00
Simon Frei
310fba4c12
lib: Return error from db.FileSet.Snapshot (fixes #7419, ref #5907) (#7424) 2021-03-07 13:43:22 +01:00
Simon Frei
767e1c6f58
lib/connections: Expose SetReadBuffer on conn passed to quic (ref #7417) (#7432) 2021-03-05 08:06:37 +01:00
Simon Frei
3d91f7c975
lib: Use counterfeiter to mock interfaces in tests (#7375) 2021-03-03 08:53:50 +01:00
greatroar
56b5352f64
all: Use crypt/rand through its buffered version, but not in benchmarks (#7420) 2021-03-02 19:17:20 +01:00
Jakob Borg
55d5e03639
lib/db: Remove Badger experiment (#7413) 2021-03-01 09:16:08 +01:00
greatroar
ffcaffa32f
lib/protocol: Optimize encrypted filename handling + make it more strict (#7408) 2021-02-27 08:57:12 +01:00
Simon Frei
0ffd80f380
lib/protocol: Alwasy return buffers to the pool (#7409) 2021-02-27 08:55:51 +01:00
wouter bolsterlee
25151b14e7
lib/api: Treat *.localhost as valid localhost addresses (#7412) (ref #4815)
This loosens the ‘is this localhost?’ check to include *.localhost host
names.

This allows for clearer (hence better) names to be used in browsers,
e.g. when accessing a remote syncthing instance ‘foo’ using a ssh port
forward, one can use foo.localhost to remind oneself which one is which.
💡 Without these changes, Syncthing shows a ‘Host check error’ when
pointing a browser at http://foo.localhost/, and with these changes, the
interface loads as usual.

The .localhost top level domain is a reserved top-level domain (RFC 2606):

> The ".localhost" TLD has traditionally been statically defined in
> host DNS implementations as having an A record pointing to the
> loop back IP address and is reserved for such use.  Any other use
> would conflict with widely deployed code which assumes this use.
> – https://tools.ietf.org/html/rfc2606

As Wikipedia puts it:

> This allows the use of these names for either documentation purposes
or in local testing scenarios. – https://en.wikipedia.org/wiki/.localhost

On Linux systems, systemd-resolved resolves *.localhost, on purpose:
https://www.freedesktop.org/software/systemd/man/systemd-resolved.service.html

See also #4815, #4816.
2021-02-27 08:52:49 +01:00
Simon Frei
fff8805ff6
all: Fix versioning path handling (#7407) 2021-02-26 12:04:05 +01:00
Simon Frei
0bf9645f2f
lib/api: Rename config insync endpoint to restart-required (#7402) 2021-02-25 09:29:44 +01:00
tomasz1986
1ed0116147
all: Fix Microsoft documentation links in code comments (#7387) 2021-02-20 14:56:45 +01:00
Simon Frei
aa1c274231
lib/fs: Consider options in case-fs caching (fixes #7371) (#7381) 2021-02-19 11:06:25 +01:00
Simon Frei
78c2844e3f
lib/scanner: Pass on errors while hashing (#7380) 2021-02-19 08:51:39 +01:00
Jakob Borg
327604719a
lib/connections: Allow QUIC with Go 1.16 (#7372) 2021-02-17 11:09:16 +01:00
Audrius Butkevicius
fb078068b4
cmd/syncthing: Add cli as a subcommand (fixes #6566, fixes #4719) (#7364)
* cmd/syncthing: Add cli as a subcommand (fixes #6566, fixes #4719)

* Hijack help

* Add comment

* Revert go.mod/go.sum
2021-02-15 18:50:53 +01:00
Simon Frei
80fc238bec
all: Automatic/disabled folder-config when receive-encrypted (#7327) 2021-02-12 22:51:29 +01:00
Simon Frei
7e4e2f3720
lib/versioner: Improve error messages (fixes #7354) (#7357) 2021-02-12 20:30:51 +01:00
Jakob Borg
55afa625fc
cmd/syncthing: Add decrypt subcommand (#7332)
This adds the `syncthing decrypt` subcommand that is used to
(offline-)decrypt or just verify the contents of an encrypted folder.
2021-02-12 08:38:43 +01:00
Jakob Borg
4f20c900d0
cmd/syncthing: Refactor command line parsing (#7330) 2021-02-10 20:35:37 +01:00
Simon Frei
d842197931
lib/model: Disable tests involving scrypt with -short (fixes #7344) (#7346) 2021-02-08 17:13:28 +01:00
Simon Frei
c0f353c0e8
lib: Do not set ModifiedBy on meta only changes (#7345) 2021-02-08 15:30:39 +01:00
Simon Frei
11e9d575c8
lib/model: Refactor folder.scanSubdirs into smaller parts (#7321) 2021-02-08 08:40:57 +01:00
Simon Frei
46bbc78e82
lib/db: Fix and improve removing entries from global (ref #6501) (#7336) 2021-02-08 08:38:41 +01:00
Jakob Borg
6db8dc33f2
lib/model: Correctly verify short read blocks (fixes #7333) (#7334)
An untrusted device will receive padded info for small blocks, and hence
sometimes request a larger block than actually exists on disk.
Previously we let this pass because we didn't have a hash to compare to
in that case and we ignored the EOF error based on that.

Now the untrusted device does pass an encrypted hash that we decrypt and
verify. This means we can't check for len(hash)==0 any more, but on the
other hand we do have a valid hash we can apply to the data we actually
read. If it matches then we don't need to worry about the read
supposedly being a bit short.
2021-02-05 16:07:21 +01:00
Jakob Borg
194501c958
lib/api: Give the config changes some more time? (#7335) 2021-02-05 15:33:37 +01:00
Simon Frei
27a34609a1
all: Failure reporting fixes (#7331) 2021-02-05 11:21:14 +01:00
Simon Frei
ffc14a77c6
all: Add configurable defaults (fixes #4224, fixes #6086) (#7131) 2021-02-04 21:10:41 +01:00
greatroar
31119ed61a
lib/ignore: Store cache timestamps as Unix ns counts (#7326) 2021-02-04 18:39:06 +01:00
Simon Frei
070bf3b776
lib/db: Report number of repaired items from checkGlobal (#7329) 2021-02-04 14:42:46 +01:00
greatroar
42917d707d
lib/scanner: Remove unused field, move WaitGroup.Add out of loop (#7323) 2021-02-03 14:25:24 +01:00
Jakob Borg
3b7a57d108
lib/protocol: Hide repeated data blocks in a given file (#7319) 2021-02-02 20:15:14 +01:00
Audrius Butkevicius
a7d9268e4d
lib/model: Make /browse endpoint return sane objects (#7306) 2021-02-01 09:27:34 +01:00
Simon Frei
052dc13487
lib/model: Correct pull progress for small files (fixes #7263) (#7316) 2021-01-31 23:40:15 +01:00
Simon Frei
249bcb3a01
lib/model: Optimize rename detection in scanner (#7315) 2021-01-31 21:02:42 +01:00
greatroar
fbe52faf49
lib/scanner: Allocate structure for final partial block (#7310)
Benchmark results on Linux/amd64, using updated benchmark for old and
new:

name        old time/op    new time/op    delta
HashFile-8    88.6ms ± 1%    88.3ms ± 1%   -0.33%  (p=0.046 n=19+19)

name        old speed      new speed      delta
HashFile-8   201MB/s ± 1%   202MB/s ± 1%   +0.33%  (p=0.044 n=19+19)

name        old alloc/op   new alloc/op   delta
HashFile-8    59.4kB ± 0%    46.1kB ± 0%  -22.47%  (p=0.000 n=14+20)

name        old allocs/op  new allocs/op  delta
HashFile-8      29.0 ± 0%      27.0 ± 0%   -6.90%  (p=0.000 n=20+20)

Co-authored-by: greatroar <@>
2021-01-28 14:23:24 +01:00
greatroar
8b86171642
lib/stun: Inline util.OnDone, comment on its purpose (#7308)
Co-authored-by: greatroar <@>
2021-01-27 19:27:00 +01:00
Simon Frei
e19d6e993d
lib/fs: Cache all real-case results (fixes #7270) (#7286) 2021-01-27 19:25:34 +01:00
greatroar
ef0473c091
lib/util, lib/svcutil: Remove unused code (#7309)
Duplicates the definition in lib/svcutil.

Co-authored-by: greatroar <@>
2021-01-27 16:33:01 +01:00
greatroar
6c3e187d1d
lib/svcutil: Simplify doneService (#7303)
OnSupervisorDone no longer allocates.

Co-authored-by: greatroar <@>
2021-01-25 16:27:17 +01:00
André Colomb
e32a516b5f
lib/model: Forget pending folders no longer announced in ClusterConfig (fixes #5187) (#7205)
* lib/db: Add ExpirePendingFolders().

Use-case is to drop any no-longer-pending folders for a specific
device when parsing its ClusterConfig message where previously offered
folders are not mentioned any more.

The timestamp in ObservedFolder is stored with only second precision,
so round to seconds here as well.  This allows calling the function
within the same second of adding or updating entries.

* lib/model: Weed out pending folders when receiving ClusterConfig.

Filter the entries by timestamp, which must be newer than or equal to
the reception time of the ClusterConfig.  For just mentioned ones,
this assumption will hold as AddOrUpdatePendingFolder() updates the
timestamp.

* lib/model, gui: Notify when one or more pending folders expired.

Introduce new event type FolderOfferCancelled and use it to trigger a
complete refreshCluster() cycle.  Listing individual entries would be
much more code and probably just as much work to answer the API
request.

* lib/model: Add comment and rename ExpirePendingFolders().

* lib/events: Rename FolderOfferCancelled to ClusterPendingChanged.

* lib/model: Reuse ClusterPendingChanged event for cleanPending()

Changing the config does not necessarily mean that the
/resut/cluster/pending endpoints need to be refreshed, but only if
something was actually removed.  Detect this and indicate it through
the ClusterPendingChanged event, which is already hooked up to requery
respective endpoints within the GUI.

No more need for a separate refreshCluster() in reaction to
ConfigSaved event or calling refreshConfig().

* lib/model: Gofmt.

* lib/db: Warn instead of info log for failed removal.

* gui: Fix pending notifications not loading on GUI start.

* lib/db: Use short device ID in log message.

* lib/db: Return list of expired folder IDs after deleting them.

* lib/model: Refactor Pending...Changed events.

* lib/model: Adjust format of removed pending folders enumeration.

Use an array of objects with device / folder ID properties, matching
the other places where it's used.

* lib/db: Drop invalid entries in RemovePendingFoldersBeforeTime().

* lib/model: Gofmt.

My local gofmt did not complain here, strangely...

* gui: Handle PendingDevicesChanged event.

Even though it currently only holds one device at a time, wrap the
contents in an array under the "added" property name.

* lib/model: Fix null values in PendingFoldersChanged removed member.

* gui: Handle PendingFoldersChanged event.

* lib/model: Simplify construction of expiredPendingList.

* lib/model: Reduce code duplication in cleanPending().

Use goto and a label for the common parts of calling the DB removal
function and building the event data part.

* lib/events, gui: Mark ...Rejected events deprecated.

Extend comments explaining the conditions when the replacement event
types are emitted.

* lib/model: Wrap removed devices in array of objects as well.

* lib/db: Use iter.Value() instead of needless db.Get(iter.Key())

* lib/db: Add comment explaining RemovePendingFoldersBeforeTime().

* lib/model: Rename fields folderID and deviceID in event data.

* lib/db: Only list actually expired IDs as removed.

Skip entries where Delete() failed as well as invalid entries that got
removed automatically.

* lib/model: Gofmt
2021-01-25 10:58:10 +00:00
greatroar
6da83ac9f5
lib/util: Remove duplicate error handling code (#7299)
This is also in lib/svcutil, and never used by clients.

Co-authored-by: greatroar <@>
2021-01-24 20:19:10 +01:00
greatroar
9c88efd55f
lib/util: Don't modify input in UniqueTrimmedStrings (#7288)
Also clarified the comment.
2021-01-16 17:39:15 +01:00
Jakob Borg
ffcb57580f
cmd/syncthing: Provide early startup for config service (ref #7188) (#7285) 2021-01-16 12:58:02 +01:00
Simon Frei
f63cdbfcfa
lib: Apply config changes sequentially (ref #5298) (#7188) 2021-01-15 15:43:34 +01:00
Simon Frei
b2d82da20d
lib/model: Pull when folder leaves error state (fixes #7280) (#7281) 2021-01-14 13:29:01 +01:00
Jakob Borg
36acaf5e21
lib/fs: Avoid blocking new caseFs creation while waiting to drop cache (fixes #7273) (#7275) 2021-01-13 17:45:29 +01:00
Jakob Borg
253049a4cc
lib/api, lib/model: Avoid contention on filesystem for DB status call (ref #7270) (#7271)
This splits the ignore getting to two methods, one that loads from disk
(the old one) and one that just returns whatever is already loaded (the
new one). The folder summary service which is just interested in stats
now uses the latter method. This means that it, and API calls that call
it, does not get blocked by folder I/O.
2021-01-12 16:25:21 +01:00
Jakob Borg
8f199e12b3
lib/fs, lib/model: Reduce lock contention on NewCaseFilesystem (fixes #7268) (#7269) 2021-01-12 16:22:21 +01:00