Commit Graph

2066 Commits

Author SHA1 Message Date
greatroar
a0fd619df3
lib/protocol: Require at least 3.125% savings from compression (#8133)
* lib/protocol: Require at least 3.125% savings from compression

The new lz4 library doesn't need its output buffer to be the maximum
size, unlike the old one (which would allocate if it weren't). It can
take a buffer that is of a smaller size and will report if compressed
data can fit inside the buffer (with a small chance of reporting a false
negative). Use that property to our advantage by requiring compressed
data to be at most n-n/32 = .96875*n bytes long for n input bytes.

* lib/protocol: Remove unused receivers

To make DeepSource happy.

* lib/protocol: Micro-optimize lz4Compress

Only write the length if compression was successful. This is a memory
write, so the compiler can't reorder it.

Only check the return value of lz4.CompressBlock.  Length-zero inputs
are always expanded by LZ4 compression (the library documents this),
so the check on len(src) isn't needed.
2022-01-24 19:36:58 +01:00
Jakob Borg
b6d1e16b4e
lib/protocol: Switch to a newer lz4 package (#8122) 2022-01-17 18:52:43 +01:00
Simon Frei
21d04b895a
lib, gui: Default ignores for new folders (fixes #7428) (#7530) 2022-01-13 23:38:21 +01:00
André Colomb
0cba3154f0
lib/model: Remove bogus fields from connections API endpoint (fixes #8103) (#8104)
* lib/model: Remove bogus fields from connections API endpoint.

Switch the returned data type for the /rest/system/connections element
"total" to use only the Statistics struct.  The other fields of the
ConnectionInfo struct are not populated and misleading.

* Lowercase JSON field names.

* lib/model: Get rid of ConnectionInfo.MarshalJSON().

It was missing the StartedAt field from the embedded Statistics
struct.  Just lowercasing the JSON attribute names can be done just as
easily with annotations.

* lib/model: Remove bogus startedAt field from totals.

Instead of using the Statistics type with one field empty, just switch
to a free-form map with the three needed fields.
2022-01-10 10:26:45 +01:00
André Colomb
5237337626
cmd/syncthing: Add --skip-port-probing (fixes #8090) (#8099)
* cmd/syncthing: Remove unnecessary function arguments.

The openGUI() function does not need a device ID to work, and there is
only one caller anyway which uses EmptyDeviceID.

The loadOrDefaultConfig() function is always called with the same
dummy values.

* cmd/syncthing: Avoid misleading info messages from monitor process.

In order to check whether panic reporting is enabled, the monitor
process utilizes the loadOrDefaultConfig() function.  In case there is
no config file yet, info messages may be logged during creation if the
config Wrapper, which is discarded immediately after.

Stop using the DefaultConfig() utility function from lib/syncthing and
directly generate a minimal config instead to avoid these.

Add comments to loadOrDefaultConfig() explaining its limited purpose.

* cmd/syncthing/generate: Always write updated config file.

Previously, an existing config file was left untouched unless either
of the --gui-user or --gui-password options was given.  Remove that
condition and simplify the checking code.

* lib/config: Factor out ProbeFreePorts().

* cmd/syncthing: Add option --skip-port-probing.

Applies to both the "generate" and "serve" subcommands, as well as the
deprecated --generate option, just as the --no-default-folder flag.
2022-01-07 11:19:17 +01:00
Gahl Saraf
cc39341eb9
lib: Fix panic due to closed event subscriptions on shutdown (#8079) 2021-12-22 20:16:21 +01:00
greatroar
eb857dbc45 lib/osutil: Use x/sys/windows for SetLowPriority 2021-11-27 15:35:07 +01:00
greatroar
286a25ae49 lib/upgrade: Use strings.Reader instead of bytes.Buffer 2021-11-27 15:35:07 +01:00
greatroar
ae70046b49 lib/protocol: Remove unused sorting boilerplate 2021-11-27 15:35:07 +01:00
greatroar
c366933416 lib/sync: Make the clock a function pointer 2021-11-27 15:35:07 +01:00
greatroar
6a9716e8a1 lib/model: Return index from deviceActivity.leastBusy
This way, we don't need a second loop over the Availabilities to remove
the selected item.
2021-11-26 12:07:43 +01:00
greatroar
8a1e54d58a lib/fs: Optimize Canonicalize
When pathSep is a constant, the compiler precomputes pathSep+pathSep and
".."+pathSep instead of emitting function calls to compute "//" and
"../". Benchmark results in lib/osutil:

name                old time/op    new time/op    delta
TraversesSymlink-8    8.86µs ± 3%    8.53µs ± 4%  -3.79%  (p=0.000 n=18+20)

name                old alloc/op   new alloc/op   delta
TraversesSymlink-8    1.06kB ± 0%    1.06kB ± 0%    ~     (all equal)

name                old allocs/op  new allocs/op  delta
TraversesSymlink-8      15.0 ± 0%      15.0 ± 0%    ~     (all equal)
2021-11-23 21:25:29 +01:00
greatroar
3e032c4da6 lib/fs: optimize Windows path checking/sanitizing
name                          old time/op    new time/op    delta
WindowsInvalidFilenameValid-8     875ns ± 1%     150ns ± 1%  -82.84%  (p=0.000 n=9+9)
WindowsInvalidFilenameNUL-8       276ns ± 4%     121ns ± 3%  -56.26%  (p=0.000 n=10+10)

name                          old alloc/op   new alloc/op   delta
WindowsInvalidFilenameValid-8     32.0B ± 0%     16.0B ± 0%  -50.00%  (p=0.000 n=10+10)
WindowsInvalidFilenameNUL-8       32.0B ± 0%     19.0B ± 0%  -40.62%  (p=0.000 n=10+10)

name                          old allocs/op  new allocs/op  delta
WindowsInvalidFilenameValid-8      2.00 ± 0%      1.00 ± 0%  -50.00%  (p=0.000 n=10+10)
WindowsInvalidFilenameNUL-8        2.00 ± 0%      2.00 ± 0%     ~     (all equal)
2021-11-23 21:25:29 +01:00
Jakob Borg
1754c93370
lib/config, lib/ignore: Write Windows line endings (fixes #7115) (#8052) 2021-11-22 09:38:24 +01:00
Jakob Borg
4b750b6dc3
all: Remove usage of deprecated io/ioutil (#7971)
As of Go 1.16 io/ioutil is deprecated. This replaces usage with the
corresponding functions in package os and package io.
2021-11-22 08:59:47 +01:00
greatroar
bf89bffb0b
lib/config: Decouple VerifyConfiguration from Committer (#7939)
... and remove 8/10 implementations, which were no-ops. This saves code
and time copying configurations.
2021-11-22 08:45:29 +01:00
Jakob Borg
e2288fe441
lib/relay: Send SNI when the address is a host name (fixes #8014) (#8015) 2021-11-22 08:31:03 +01:00
greatroar
8265dac127
lib/nat: Fix race condition in Mapping (#8042)
The locking protocol in nat.Mapping was racy:

* Mapping.addressMap RLock'd, but then returned a map shared between
  caller and Mapping, so the lock didn't do anything.

* Operations inside Service.{verifyExistingMappings,acquireNewMappings}
  would lock the map for every update, but that means callers to
  Mapping.ExternalAddresses can be looping over the map while the
  Service methods are concurrently modifying it. When the Go runtime
  detects that happening, it panics.

* Mapping.expires was read and updated without locking.

The Service methods now lock the map once and release the lock only when
done.

Also, subscribers no longer get the added and removed addresses, because
none of them were using the information. This was changed for a previous
attempt to retain the fine-grained locking and not reverted because it
simplifies the code.
2021-11-22 08:29:44 +01:00
André Colomb
100870e142
cmd/syncthing: Implement generate as a subcommand with optional API credential setting (fixes #8021) (#8043)
Accept a subcommand as an alternative to the --generate option.  It
accepts a custom config directory through either the --home or
--config options, using the default location if neither is given.

Add the options --gui-user and --gui-password to "generate", but not
the "serve --generate" option form.  If either is given, an existing
config will not abort the command, but rather load, modify and save it
with the new credentials.  The password can be read from standard
input by passing only a single dash as argument.

Config modification is skipped if the value matches what's already in
the config.

* cmd/syncthing: Utilize lib/locations package in generate().
Instead of manually joining paths with "magic" file names, get them
from the centralized locations helper lib.

* cmd/syncthing: Simplify logging for --generate option.
Visible change: No more timestamp prefixes.
2021-11-18 22:57:59 +01:00
Jakob Borg
12fb7f2a0a lib/model: Correct "reverting folder" log entry 2021-11-17 12:52:10 +01:00
Jakob Borg
f1bf4d899a lib/model: Correct handling of fakefs cache
We looked under one cache key, then stored under another...
2021-11-17 12:52:10 +01:00
Simon Frei
591e4d8af1
gui, lib: Fix tracking deleted locally-changed on encrypted (fixes #7715) (#7726) 2021-11-10 09:46:21 +01:00
André Colomb
dec6f80d2b
lib/config: Move the bcrypt password hashing to GUIConfiguration (#8028)
What hash is used to store the password should ideally be an
implementation detail, so that every user of the GUIConfiguration
object automatically agrees on how to handle it.  That is currently
distribututed over the confighandler.go and api_auth.go files, plus
tests.

Add the SetHasedPassword() / CompareHashedPassword() API to keep the
hashing method encapsulated.  Add a separate test for it and adjust
other users and tests.  Remove all deprecated imports of the bcrypt
package.
2021-11-08 13:32:04 +01:00
André Colomb
ec8a748514
lib/syncthing: Clean up / refactor LoadOrGenerateCertificate() utility function. (#8025)
LoadOrGenerateCertificate() takes two file path arguments, but then
uses the locations package to determine the actual path.  Fix that
with a minimally invasive change, by using the arguments instead.
Factor out GenerateCertificate().

The only caller of this function is cmd/syncthing, which passes the
same values, so this is technically a no-op.

* lib/tlsutil: Make storing generated certificate optional.  Avoid
  temporary cert and key files in tests, keep cert in memory.
2021-11-07 23:59:48 +01:00
greatroar
db15e52743
lib/api: http.Request.BasicAuth instead of custom code (#8039) 2021-11-06 12:38:08 +01:00
André Colomb
41bfb7a330
Normalize CLI options to always use two dashes. (#8037)
Consistently use double dashes and fix typos -conf, -data-dir and
-verify.

Applies also to tests running the syncthing binary for consistency.

* Fix mismatched option name --conf in cli subcommand.

According to the source code comments, the cli option flags should
mirror those from the serve subcommand where applicable.  That one is
actually called --config though.

* cli: Fix help text option placeholders.

The urfave/cli package uses the Value field of StringFlag to provide a
default value, not to name the placeholder.  That is instead done with
backticks around some part of the Usage field.

* cli: Add missing --data flag in subcommand help text.

The urfave/cli based option parsing uses a fake flags collection to
generate help texts matching the used global options.  But the --data
option was omitted from it, although it is definitely required when
using --config as well.  Note that it cannot just be ignored, as some
debug stuff actually uses the DB:

syncthing cli --data=/bar --config=/foo debug index dump
2021-11-04 08:42:55 +01:00
greatroar
807a6b1022
lib/model: Optimize jobQueue performance and memory use (#8023)
By truncating time.Time to an int64 nanosecond count, we lose the
ability to precisely order timestamps before 1678 or after 2262, but we
gain (linux/amd64, Go 1.17.1):

name                      old time/op    new time/op    delta
JobQueuePushPopDone10k-8    2.85ms ± 5%    2.29ms ± 2%  -19.80%  (p=0.000 n=20+18)
JobQueueBump-8              34.0µs ± 1%    29.8µs ± 1%  -12.35%  (p=0.000 n=19+19)

name                      old alloc/op   new alloc/op   delta
JobQueuePushPopDone10k-8    2.56MB ± 0%    1.76MB ± 0%  -31.31%  (p=0.000 n=18+13)

name                      old allocs/op  new allocs/op  delta
JobQueuePushPopDone10k-8      23.0 ± 0%      23.0 ± 0%     ~     (all equal)

Results for BenchmarkJobQueueBump are with the fixed version, which no
longer depends on b.N for the amount of work performed. rand.Rand.Intn
is cheap at ~10ns per iteration.
2021-10-29 20:20:46 +02:00
Tomasz Wilczyński
296cc1bca2 lib/model: Limit the number of default hashers on Android (ref #2220)
Like Windows and Mac, Android is also an interactive operating system.
On top of that, it usually runs on much slower hardware than the other
two. Because of that, it makes sense to limit the number of hashes used
by default there too.

Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
2021-10-29 10:06:52 +02:00
Simon Frei
951b058952 lib/model: Set mod. time after writing trailer in shortcut (ref #7992) 2021-10-24 16:50:30 +02:00
greatroar
8f8e8a9285 lib/protocol: Simplify codeToError, errorToCode
Also be explicit about the fact that ErrNoError is nil. That name isn't
used anywhere outside this file.
2021-10-22 18:40:40 +02:00
greatroar
46082f194c lib/protocol: Eliminate nativeModel on Unix 2021-10-22 18:40:40 +02:00
Simon Frei
517667c590
lib/model: Pull when a new connection is established (fixes #8012) (#8013) 2021-10-20 18:55:22 +02:00
greatroar
7c292cc812
lib/connections: Fix and optimize registry (#7996)
Registry.Get used a full sort to get the minimum of a list, and the sort
was broken because util.AddressUnspecifiedLess assumed it could find out
whether an address is IPv4 or IPv6 from its Network method. However,
net.(TCP|UDP)Addr.Network always returns "tcp"/"udp".
2021-10-06 10:52:51 +02:00
Simon Frei
c94b797f00
lib/protocol: Preserve sequence decrypting fileinfos (fixes #7994) (#7995) 2021-10-06 10:26:54 +02:00
Simon Frei
708a5c2070
lib/model: Write trailer when shortcutting on recv-enc (fixes #7991) (#7992) 2021-10-05 11:23:27 +02:00
Jakob Borg
92eaf52c21 lib/protocol: Test to lock down encryption key derivation
So that we don't inadvertently change how these things work.
2021-10-04 14:36:28 +02:00
tomasz1986
8d8f331a4a
lib/config: Always set mtime window to 2 on Android unless ext detected (ref #7963) (#7966)
The current detection is flawed, because it looks for a few specific
file systems like "msdos" or "fat" to set the mtime window, while in
reality Android seems to report names like "fuseblk", which can stand
for fat, ext4, or even f2fs.

At the moment, we set the mtime window only for a few known names used
for the fat filesystem. With this change, we take a safer approach of
always setting the time window unless we explicitly detect file systems
like ext2/ext3/ex4, which are known not to experience issues with moving
timestamps on Android.

Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
2021-09-27 21:29:51 +02:00
greatroar
198028d627
lib/rand: Optimizations (#7964)
rand.secureSource.Uint64 no longer allocates. rand.String uses a
strings.Builder. Benchmark results on linux/amd64:

name            old time/op    new time/op    delta
SecureSource-8    69.1ns ± 3%    51.7ns ± 3%   -25.21%  (p=0.000 n=20+10)
String-8          2.66µs ± 2%    1.95µs ± 1%   -26.61%  (p=0.000 n=10+10)

name            old alloc/op   new alloc/op   delta
SecureSource-8     8.00B ± 0%     0.00B       -100.00%  (p=0.000 n=20+10)
String-8            288B ± 0%       32B ± 0%   -88.89%  (p=0.000 n=10+10)

name            old allocs/op  new allocs/op  delta
SecureSource-8      1.00 ± 0%      0.00       -100.00%  (p=0.000 n=20+10)
String-8            33.0 ± 0%       1.0 ± 0%   -96.97%  (p=0.000 n=10+10)
2021-09-26 12:15:39 +02:00
Jakob Borg
73c5184518
build: Update all dependencies (#7968)
Also add a script step for how to do this, as frankly I forget each time
what the canonical way is.

`go run build.go update-deps`
2021-09-26 12:08:23 +02:00
greatroar
f96c211198
lib/db: Replace SipHash with hash/maphash (#7962) 2021-09-24 21:26:07 +02:00
greatroar
a4489dec30
lib/tlsutil: Allocate UnionedConnection in one go (#7912) 2021-09-21 08:40:34 +02:00
Simon Frei
721cd740d8
lib/model: Don't reset db while folder is running (fixes #7935) (#7936) 2021-09-11 17:14:47 +02:00
greatroar
de719ac409
lib/db: Inline sync.Once in releaser (#7938)
Having a pointer to a Once suggests that it is shared with other
objects, but it never is.
2021-09-10 09:58:17 +02:00
jtagcat
0ffa7f3f57
all: Clarify 'Cleaning data for folder' log message (#7937)
Instead of data (could be read as user data), use metadata.
2021-09-10 09:56:44 +02:00
greatroar
7fa141ea39
all: Unused args, retvals, assignments (#7926) 2021-09-08 00:11:16 +02:00
greatroar
13196ddd92 lib/relay/protocol: Merge two Sprintf calls 2021-09-06 15:30:56 +02:00
greatroar
eafb40460d lib/discover, lib/protocol: Buffer allocation 2021-09-06 15:30:56 +02:00
Simon Frei
4e2a9bb139
lib/model: Remove encryption pw on folder stop (#7925) 2021-09-03 09:54:47 +02:00
greatroar
3b2239357f
lib/model: Add missing error assignment + Remove unused argument (#7922) 2021-09-01 22:03:06 +02:00
Ross Smith II
7501bee430
lib/model: Typo in folder_recvonly.go (#7919) 2021-09-01 07:41:18 +02:00
greatroar
ed98039aa5
lib/fs: Optimize TempName + some cosmetic changes (#7911) 2021-08-29 10:47:53 +02:00
Jakob Borg
2816780b52
lib/api: Set "Secure" on session cookies served over HTTPS (ref #7399) (#7907)
So that it does not unnecessarily leak over clear text connections.
2021-08-27 17:56:54 +02:00
xjtdy888
48796a1b60
lib/ur: Fix panic build goroutines for failures (#7903) 2021-08-25 07:16:55 +02:00
Simon Frei
70a840d3d5
lib/connections: Remove future go build constraints on quic (#7898) 2021-08-19 21:05:28 +02:00
Jakob Borg
d47745a86b
all: Update build constraints to Go 1.17 style (#7894) 2021-08-17 10:10:41 +02:00
Simon Frei
0fe72e6fc5
lib/model: Schedule pull after revert on recv-enc (#7892) 2021-08-17 09:24:10 +02:00
Simon Frei
c025e76f30
lib/model: Do not wait for scan when setting ignores (fixes #7893) (#7891) 2021-08-17 09:23:33 +02:00
Simon Frei
e1bf1e672e
lib/ur: Fix panic getting goroutines for failures (ref #7785) (#7890) 2021-08-16 22:47:05 +02:00
Simon Frei
db302b15ea
lib/syncthing: Set system timezone on android (#7878) 2021-08-09 09:27:14 +02:00
Simon Frei
952f3ffb0c
lib/db/backend: Prevent panic in errors.As (#7873) 2021-08-05 11:25:20 +01:00
Audrius Butkevicius
fe77fac23f
Update quic to 0.22.0 to support RFC9000, enable batch reads (#7862)
* Update quic to 0.22.0 to support RFC9000, enable batch reads
* Remove wrappers that are not needed anymore
2021-08-05 05:44:22 +02:00
Simon Frei
e61091d240
lib/stun: Prevent nil deref when naming service (#7872) 2021-08-05 00:04:22 +01:00
Simon Frei
50aacdf1f0
lib/protocol: Ensure correct blocksize on enc. fileinfo (ref #7861) (#7870) 2021-08-04 23:12:01 +02:00
Simon Frei
e56e8b7aa1
lib/model: Don't consider hashes pulling on recv-enc (#7869) 2021-08-04 11:10:10 +02:00
tomasz1986
5a1f6cb813
lib/fs: Improve case conflict error message (fixes #7827) (#7829) 2021-08-01 22:44:49 +02:00
greatroar
37d0ba1660
lib/watchaggregator: Replace counter map by two integers (#7856) 2021-08-01 12:30:20 +02:00
Jonathan
0836439256
lib/model: Fix config deadlock when deleting a paused folder (#7854) 2021-07-30 14:41:00 +02:00
Simon Frei
67b18569cf
all: Send deadlocks as failures, crash only as a last resort (#7785) 2021-07-27 21:27:52 +02:00
Simon Frei
dc0dd09e93
lib/model: Don't try to delete deleted item on revert (#7843) 2021-07-23 14:26:20 +02:00
Simon Frei
eeb7091180
lib/model: Missing fmut-lock on encryption failures (#7841) 2021-07-22 11:15:25 +02:00
greatroar
dc38e6ae88
lib/relay/client: Stricter typing and remove unused code (#7819) 2021-07-21 09:49:09 +02:00
greatroar
1e21042138
lib/connections: switch statement to get the QUIC network (#7816) 2021-07-10 13:53:51 +02:00
Chih-Hsuan Yen
11c57b9097
lib/connections: Resolve IPv6 for quic6:// peers (fixes #7809) (#7810)
Before this patch, IPv4-compatible addresses (::ffff:aaa.bbb.ccc.ddd)
may be used if a quic6://some.domain:port is specified and both IPv4 and
IPv6 addresses exist for that domain name.
2021-07-05 13:19:56 +02:00
Simon Frei
1921533c4c
lib/connections: Fully dial resumed devices (#7798) 2021-07-03 18:26:55 +02:00
Simon Frei
89e762fd6e
lib/model: Prevent folder-type change from/to encrypted (fixes #7704) (#7796) 2021-07-03 13:47:04 +02:00
Simon Frei
a63d3ee625
lib/model: Scan removed dirs when reverting recv-enc (fixes #7706) (#7797) 2021-07-03 13:46:24 +02:00
greatroar
bd363fe0b7
lib/protocol: Write uncompressible messages uncompressed (#7790) 2021-06-27 17:59:30 +02:00
Simon Frei
445a82f120
lib/model: Compare all items with global on scan (fixes #7740) (#7791) 2021-06-27 08:48:54 +02:00
Simon Frei
69ce121267
lib/db: Missing event-logger in write-transaction (#7793) 2021-06-27 08:43:49 +02:00
Simon Frei
08e3cd1cce
lib/fs: Set expiry after DirNames in case-fs (#7794) 2021-06-27 08:30:02 +02:00
Simon Frei
c78fa42f31
lib/connections: Dial devices in parallel (#7783) 2021-06-25 11:38:04 +02:00
Jakob Borg
993a3ebe73
lib/api: Always include usage reporting data in support bundle (#7786) 2021-06-24 22:00:14 +04:00
Simon Frei
400d62c1e6
lib/connections: Missed map init in nextDialAt (ref #7753) (#7778) 2021-06-17 21:13:57 +04:00
Simon Frei
857caf3637
lib/connections: Trigger dialer when connection gets closed (#7753) 2021-06-17 13:57:44 +02:00
Simon Frei
aeca1fb575
lib/db: Check if sequences change when repairing metadata (#7770) 2021-06-17 13:53:39 +02:00
Simon Frei
ac2988a485
gui, lib: Handle pw adding remote encrypted folder (fixes #7705) (#7772) 2021-06-17 13:53:02 +02:00
Simon Frei
23a0e18292
lib/db: Fix accounting bug when dropping indexes (#7774) 2021-06-17 10:15:11 +02:00
Simon Frei
aa6c55dec1
lib/model: Remove bogus failureevent when restarting folder (#7773) 2021-06-17 08:57:24 +02:00
Simon Frei
1e7a3997e3
lib/db, lib/model: Improve error handling on pending items (#7754) 2021-06-09 13:35:17 +02:00
Anur
e7f8538e4d
lib/fs: Add bitmasks for Darwin to handle change to empty files (fixes #7731) (#7756) 2021-06-09 12:57:06 +02:00
Simon Frei
1a22689328
lib/db: Add failure reports to failures iterating over hashes (#7755) 2021-06-07 23:10:35 +02:00
Jakob Borg
ce65aea0ab
lib/db: Use a more concurrent GC (fixes #7722) (#7750)
This changes the GC mechanism so that the first pass (which reads all
FileInfos to populate bloom filters with block & version hashes) can
happen concurrently with normal database operations.

The big gcMut still exists, and we grab it temporarily to block all
other modifications while we set up the bloom filters. We then release
the lock and let other things happen, with those other things also
updating the bloom filters as required. Once the first phase is done we
again grab the gcMut, knowing that we are the sole modifier of the
database, and do the cleanup.

I also removed the final compaction step.
2021-06-07 10:52:06 +02:00
André Colomb
45edad867c
all: Allow dismissing pending devices / folders without ignoring (fixes #7700) (#7712) 2021-06-07 10:29:24 +02:00
André Colomb
ea0a408849
gui: Modal dialog for listeners and discovery status (#7539) 2021-06-07 09:08:44 +02:00
Simon Frei
18592af993
lib/model: Fix wrongly hardcoded arguments in test helper (#7749) 2021-06-05 17:01:23 +02:00
Simon Frei
b1e0e7b923
lib/model: Fix indexhandling for new folders paused on remote (#7747) 2021-06-05 16:27:15 +02:00
Simon Frei
6494a9332d
lib/model: Fix test introduced in #7714 failing due to #7689 (#7745) 2021-06-04 15:32:47 +02:00
Simon Frei
41baccb85d
lib/model: Fix passwords on receive-enc needing token (ref #7518) (#7739) 2021-06-03 15:39:49 +02:00
Simon Frei
52eb7392c4
lib/api, lib/config: Apply defaults before deserializing json (#7690) 2021-06-03 15:09:35 +02:00
Simon Frei
855c53ad02
lib/model: Fix reverting when version has only our own ID (fixes #7708) (#7714) 2021-06-03 15:08:56 +02:00
Simon Frei
004eded398
lib/model: Don't share with introduced device if encrypted (fixes #7724) (#7734) 2021-06-03 15:02:57 +02:00
Simon Frei
df48276300
lib/model: Ensure indexes are only received after checking IDs (ref #7649) (#7689) 2021-06-03 14:58:50 +02:00
greatroar
95c9561e97
lib/db: Clean up Timer and wait for logging before return in GC (#7720) 2021-05-31 09:50:21 +02:00
Simon Frei
fcb19518c7
build, lib/model: Add flag to run tests without -short and fix failure (#7716) 2021-05-28 22:20:18 +02:00
Simon Frei
22e09334ec
lib/model: Fix incoming request on receive-enc (fixes #7699) (#7702) 2021-05-22 21:38:49 +02:00
Simon Frei
58592e3ef1
lib/db: Add logging for GC (#7707) 2021-05-22 21:36:43 +02:00
Simon Frei
0126188ba7
lib/config: Set DisableTempIndexes to true on receive-encrypted (#7701) 2021-05-20 22:33:23 +02:00
Simon Frei
5bdb6798a9
all: Regenerate proto (#7696) 2021-05-19 13:30:20 +02:00
greatroar
0e52ce830a
lib/fs: Fix UnicodeLowercaseNormalized on lowercase NFD (#7692)
Co-authored-by: greatroar <@>
2021-05-17 20:43:07 +02:00
Jakob Borg
97437cad64
lib/fs: Ignore normalization differences in case insensitive lookup (fixes #7677) (#7678) 2021-05-17 12:35:03 +02:00
Simon Frei
5b90a98650
lib/model: Fix addFakeConn and other test improvements (#7684) 2021-05-16 17:23:27 +02:00
Simon Frei
93a02c677e
lib/scanner: Do not drop all not-exist-errors and debug logging (#7675) 2021-05-15 11:51:35 +02:00
Simon Frei
0d054f9b64
lib/model: Don't use empty folder cfg for index sender (fixes #7649) (#7671) 2021-05-15 11:13:39 +02:00
Audrius Butkevicius
1107f6eb5f
lib/connections: Reduce default quic redial interval (fixes #7471) (#7672)
* lib/connections: Reduce default quic redial interval (fixes #7471)

* Update quic_dial.go
2021-05-14 14:26:02 +01:00
Simon Frei
c494ced21f
lib/connections: Actually remove listenerSupervisor (ref #7644) (#7663) 2021-05-11 14:35:13 +02:00
Audrius Butkevicius
aca1b45e93
lib/connections: Update pfilter to pick up bugfix/oob stuff, support OOB connections (fixes #7636) (#7654) 2021-05-11 07:59:56 +02:00
Simon Frei
5cb2a10138
lib/model: Improve encryption cluster-config errors (#7658) 2021-05-11 07:55:44 +02:00
Audrius Butkevicius
411796606c
lib/connections: Correct service termination order (#7657) 2021-05-10 22:29:27 +02:00
Simon Frei
1a9b54c9fa
lib/connections: Separate listener supervisors and lower backoff time (#7644) 2021-05-10 22:26:51 +02:00
Simon Frei
c7f4f15272
lib/relay, lib/svcutil: Improve service logging (fixes #7580) (#7647) 2021-05-10 22:26:25 +02:00
Simon Frei
713527facf
all: Refactor relay invitations (#7646) 2021-05-10 22:25:43 +02:00
Simon Frei
6e662dc9fc
lib/suture: Use ServeBackground to start main supervisor (#7626) 2021-05-10 16:50:45 +02:00
Audrius Butkevicius
eb178caf3a
lib/connections: Add connection benchmarks, allow binding to port zero addresses (#7648)
* Add connbench

* Refactor port fixup

* More cleanup

* touch for build

Co-authored-by: Jakob Borg <jakob@kastelo.net>
2021-05-10 15:44:47 +01:00
Simon Frei
6157c766de
lib/connections: Correct comments on quic wrapper type (#7652) 2021-05-09 19:15:10 +01:00
Audrius Butkevicius
745cd4744a
lib/connections: Revert add more methods to the quic conn wrapper (#7651)
This reverts commit faf15b4567.
2021-05-09 19:43:16 +02:00
Simon Frei
faf15b4567
lib/connections: Add more methods to the quic conn wrapper (#7643) 2021-05-09 12:45:08 +01:00
greatroar
3746c899b7
build: List go:generate tools in tools.go (#7599) 2021-05-08 12:52:06 +02:00
Audrius Butkevicius
87a0eecc31
lib/fs, lib/api, lib/model: Expose mtime remappings as part of /db/file (#7624)
* lib/fs, lib/api, lib/model: Expose mtime remappings as part of /db/file

* Fix wrong error returned by CLI

* Gofmt

* Better names

* Review comments

* Review comments
2021-05-03 11:28:25 +01:00
Simon Frei
75b58eb480
lib/fs: Watch attrib. changes on inotify for mod. time (#7623)
* lib/fs: Watch attrib. changes on inotify for mod. time

* fix bsds (real) and darwin (test only)
2021-05-02 16:45:44 +02:00
Simon Frei
dd39556759
lib: Revert needing invalid files (fixes #7608, ref #7476) (#7609) 2021-04-29 22:01:46 +02:00
Simon Frei
1f30383866
lib/model: Remove path from enc errors and report only once (#7610) 2021-04-29 19:21:07 +02:00
Gahl Saraf
66662cd678
Trigger connection loop on config device addition (fixes #7600) (#7604)
* Trigger connection loop on config device addition (fixes #7600)

* Also check for device address equality

* Move EqualStrings from api_test to utils, and use in connections/service.go

* Make sure CommitConfiguration cannot block due on the deviceAddressesChanged channel

* Update lib/connections/service.go

Co-authored-by: Jakob Borg <jakob@kastelo.net>
2021-04-26 21:13:59 +01:00
Simon Frei
8734fa65fc
lib/model: Handle invalid needed items on send-only (ref #7476) (#7596) 2021-04-26 15:36:51 +02:00
Jakob Borg
74823e81e9
all: Deprecate TLS 1.2 on sync connections (fixes #7594) (#7598)
This makes us use TLS 1.3+ on sync connections by default. A new option
`insecureAllowOldTLSVersions` exists to allow communication with TLS
1.2-only clients (roughly Syncthing 1.2.2 and older). Even with that
option set you get a slightly simplified setup, with the cipher suite
order fixed instead of auto detected.
2021-04-26 10:04:35 +02:00
Jakob Borg
e4c995a321
lib/model: Don't remove non-empty .stfolder (#7578) 2021-04-15 18:13:35 +02:00
André Colomb
130d14cec9
api: Log API authorization failures. (#7575) 2021-04-15 07:33:02 +02:00
Simon Frei
15d2dc3a4f
lib/connections: Add SyscallConn() to quic conn (fixes #7551) (#7570) 2021-04-13 12:59:58 +01:00
André Colomb
f6df1a760d
lib/api: Log the remote address on login attempts (#7560)
This enables usage of the audit log to e.g. automatically block remote
addresses from connecting after repeated login failures.
2021-04-13 10:14:44 +02:00
Simon Frei
f71fcd440a
all: Remove untrusted feature flag (fixes #109) (#7567)
No longer hide the web UI controls for the new untrusted/encrypted
device feature. Testing hasn't been very widespread, but there has been
some and quite a few bugs have been caught and fixed. I believe its time
to not hide it anymore, and cautiously recommend usage. E.g. mention
that the feature hasn't been widely used yet and anyone using it is an
early adopter, but drop the bit about not using it with production data.
We can maybe stress the need for backups in general and especially
using this.
2021-04-13 10:12:56 +02:00
Simon Frei
1a00ea7c6e
lib: Prevent using protocol method with native path (fixes #7557) (#7563) 2021-04-11 15:29:43 +02:00
Simon Frei
ec0a66c75b
lib/db, lib/model: Refactor removing expired pending folders (#7537) 2021-04-11 15:24:08 +02:00
Jakob Borg
8e38ecdeb2
Merge pull request from GHSA-x462-89pf-6r5h 2021-04-06 08:00:00 +02:00
Simon Frei
f30f9c50f8
lib/db: Handle indirection error repairing sequences (fixes #7026) (#7525) 2021-04-05 10:24:16 +02:00
Simon Frei
0dcd9794d4
lib/protocol: Deterministic encrypted version (fixes #7533) (#7538) 2021-03-31 08:59:15 +02:00
André Colomb
fb2d85b9d5
lib/db: Fix comparison of pending folder timestamps (fixes #7532) (#7535) 2021-03-30 21:24:53 +02:00
Simon Frei
7f0d4f6ba8
lib/db: Don't panic debugging an inexistent file (#7534) 2021-03-30 20:06:01 +02:00
Simon Frei
0d7a77ba85
lib/model: Reset conn when enc token is missing (fixes #7198) (#7518) 2021-03-23 10:38:40 +01:00
Simon Frei
924b96856f
lib: Handle adding enc folders on an existing conn (fixes #7509) (#7510) 2021-03-22 21:50:19 +01:00
Simon Frei
6b25eb2e79
lib/ur: Prevent panic when blocksResult is nil (ref #7495) (#7515) 2021-03-22 15:07:41 +01:00
Simon Frei
bc08a951f1
lib/model: Encrypted fileinfo trailer needs to be in wire format (#7505) 2021-03-21 10:34:08 +01:00
Lars Lehtonen
a87c5515bd
lib/model: Error check in test (#7508) 2021-03-21 10:32:17 +01:00
Jakob Borg
ebcd22b02b
lib/osutil: Fix raising max FDs on macOS (#7507)
There was a logic mistake, so the limit in question wasn't used. On my
macOS this doesn't seem to matter, the hard limit returned is 2^63-1 and
setting the soft limit to that works. However I'm assuming that's not
the case for older macOSes since it was so nicely documented, so we
should still have this working. (10240 FDs should be enough for
anybody.)
2021-03-20 16:32:36 +01:00
Audrius Butkevicius
4b02b7e6f1
lib/connections: Silence "connected to myself" messages. (#7500) 2021-03-17 23:53:20 +01:00
Jakob Borg
fdd823d2cb lib/osutil: Remove unused code 2021-03-17 23:18:07 +01:00
Jakob Borg
8ef504f745
all: Simplify some method calls (#7499)
strings.Replace(a, b, c, -1) -> strings.ReplaceAll(a, b, c)

(Go 1.12) and who knows what was up with that dialQueue.Sort() thing.
2021-03-17 23:12:26 +01:00
deepsource-autofix[bot]
6c573a5762
Remove unnecessary guard around delete (#7496)
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>
2021-03-17 22:23:36 +01:00
Jakob Borg
3ac858b150
all: Remove miscellaneous vestigial code (#7495) 2021-03-17 22:23:12 +01:00
Jakob Borg
f4372710bf
all: Remove crypto/md5 (#7493)
This is a mostly pointless change to make security scanners and static
analysis tools happy, as they all hate seeing md5. None of our md5 uses
were security relevant, but still. Only visible effect of this change is
that our temp file names for very long file names become slightly longer
than they were previously...
2021-03-17 22:22:49 +01:00
Jakob Borg
f39477bbd5
lib/api: Missing error handling in API delete-device (#7494) 2021-03-17 22:08:44 +01:00
Jakob Borg
6e5514419d
lib/db: Fix some omitted error checks, unused variable (#7489) 2021-03-17 21:41:07 +01:00
Jakob Borg
81484699db
lib/model: Actually break puller loop on context cancel (#7492)
Current break does nothing (breaks the select).
2021-03-17 21:34:52 +01:00
deepsource-autofix[bot]
0930bccf88
cmd/ursrv, lib/scanner: Remove unnecessary slicing of slices (#7491)
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>
2021-03-17 21:04:36 +01:00
Jakob Borg
e321bd3941
lib/*/auto: Add noassets files (#7490)
This adds a couple of dummy asset files protected by the "noassets"
build tag. The purpose is that it should be possible for, for example,
CI tools and static analysis things to compile and analyze the source
tree without our custom asset generation step. Also makes `go test -tags
noassets ./...` work without building assets first.
2021-03-17 21:03:35 +01:00
Simon Frei
4b02937862
lib/model: Add missing lock on availability func (#7487) 2021-03-17 20:46:13 +01:00
Simon Frei
3f2b584c4e
lib/model: Don't use ignore patterns for recv-enc folders (fixes #7469) (#7472) 2021-03-16 15:04:11 +01:00
Simon Frei
d2d4fcc1df
lib/protocol: Improve messages when an error occurs receiving (ref #7466) (#7470) 2021-03-15 19:14:09 +01:00
Simon Frei
273ee09925
lib/db, lib/model: Allow needing invalid files (fixes #7474) (#7476) 2021-03-15 07:58:01 +01:00
Simon Frei
f80ee472c2
lib/protocol: Set invalid flag on encrypted file infos (fixes #7466) (#7467) 2021-03-13 16:57:36 +01:00
Simon Frei
97a8777d03
lib/fs: Check both old and new path when renaming (fixes #7426) (#7463) 2021-03-12 21:15:50 +01:00
Simon Frei
8a4c00d82e
lib/model: Send failure report on CC encryption check error (#7460) 2021-03-12 12:21:54 +01:00
Simon Frei
31f859e909
lib/model: Return correct error in puller-iteration (ref #7424) (#7461) 2021-03-12 12:21:28 +01:00
Jakob Borg
4d979a1ce9
all: Truncate some timestamps (fixes #7457) (#7459)
This truncates times meant for API consumption to second precision,
where fractions won't typically matter or add any value. Exception to
this is timestamps on logs and events, and of course I'm not touching
things like file metadata.

I'm not 100% certain this is an exhaustive change, but it's the things I
found by grepping and following the breadcrumbs from lib/api...

I also considered general-but-ugly solutions, like having the API
serializer itself do reflection magic or even regexps on returned
objects, but decided against it because aurgh...
2021-03-12 10:35:10 +01:00
Simon Frei
4465cdf8bc
lib/api: Fix body of renamed config/restart-required endpoint (ref #7402) (#7453) 2021-03-11 15:54:05 +01:00
Simon Frei
3938b61c3f
lib/fs: Expose fs option on interface (fixes #7385, ref #7381) (#7389) 2021-03-11 15:23:56 +01:00
Jakob Borg
cdef503db6
all: Make config.Wrapper an actual suture.Service (fixes #7451) (#7452) 2021-03-11 14:51:00 +01:00
Jakob Borg
df08984a58
lib/api: Sanitize names used in certificates (fixes #7434) (#7435) 2021-03-11 13:15:03 +01:00
Simon Frei
9a001051d6
cmd/ursrv, lib/ur: Collect and present encryption usage (#7448) 2021-03-10 22:26:56 +00:00
greatroar
c00520281b
lib/protocol: Optimize FileKey (#7440) 2021-03-07 18:44:21 +01:00
Simon Frei
310fba4c12
lib: Return error from db.FileSet.Snapshot (fixes #7419, ref #5907) (#7424) 2021-03-07 13:43:22 +01:00
Simon Frei
767e1c6f58
lib/connections: Expose SetReadBuffer on conn passed to quic (ref #7417) (#7432) 2021-03-05 08:06:37 +01:00
Simon Frei
3d91f7c975
lib: Use counterfeiter to mock interfaces in tests (#7375) 2021-03-03 08:53:50 +01:00
greatroar
56b5352f64
all: Use crypt/rand through its buffered version, but not in benchmarks (#7420) 2021-03-02 19:17:20 +01:00
Jakob Borg
55d5e03639
lib/db: Remove Badger experiment (#7413) 2021-03-01 09:16:08 +01:00
greatroar
ffcaffa32f
lib/protocol: Optimize encrypted filename handling + make it more strict (#7408) 2021-02-27 08:57:12 +01:00
Simon Frei
0ffd80f380
lib/protocol: Alwasy return buffers to the pool (#7409) 2021-02-27 08:55:51 +01:00
wouter bolsterlee
25151b14e7
lib/api: Treat *.localhost as valid localhost addresses (#7412) (ref #4815)
This loosens the ‘is this localhost?’ check to include *.localhost host
names.

This allows for clearer (hence better) names to be used in browsers,
e.g. when accessing a remote syncthing instance ‘foo’ using a ssh port
forward, one can use foo.localhost to remind oneself which one is which.
💡 Without these changes, Syncthing shows a ‘Host check error’ when
pointing a browser at http://foo.localhost/, and with these changes, the
interface loads as usual.

The .localhost top level domain is a reserved top-level domain (RFC 2606):

> The ".localhost" TLD has traditionally been statically defined in
> host DNS implementations as having an A record pointing to the
> loop back IP address and is reserved for such use.  Any other use
> would conflict with widely deployed code which assumes this use.
> – https://tools.ietf.org/html/rfc2606

As Wikipedia puts it:

> This allows the use of these names for either documentation purposes
or in local testing scenarios. – https://en.wikipedia.org/wiki/.localhost

On Linux systems, systemd-resolved resolves *.localhost, on purpose:
https://www.freedesktop.org/software/systemd/man/systemd-resolved.service.html

See also #4815, #4816.
2021-02-27 08:52:49 +01:00
Simon Frei
fff8805ff6
all: Fix versioning path handling (#7407) 2021-02-26 12:04:05 +01:00
Simon Frei
0bf9645f2f
lib/api: Rename config insync endpoint to restart-required (#7402) 2021-02-25 09:29:44 +01:00
tomasz1986
1ed0116147
all: Fix Microsoft documentation links in code comments (#7387) 2021-02-20 14:56:45 +01:00
Simon Frei
aa1c274231
lib/fs: Consider options in case-fs caching (fixes #7371) (#7381) 2021-02-19 11:06:25 +01:00
Simon Frei
78c2844e3f
lib/scanner: Pass on errors while hashing (#7380) 2021-02-19 08:51:39 +01:00
Jakob Borg
327604719a
lib/connections: Allow QUIC with Go 1.16 (#7372) 2021-02-17 11:09:16 +01:00
Audrius Butkevicius
fb078068b4
cmd/syncthing: Add cli as a subcommand (fixes #6566, fixes #4719) (#7364)
* cmd/syncthing: Add cli as a subcommand (fixes #6566, fixes #4719)

* Hijack help

* Add comment

* Revert go.mod/go.sum
2021-02-15 18:50:53 +01:00
Simon Frei
80fc238bec
all: Automatic/disabled folder-config when receive-encrypted (#7327) 2021-02-12 22:51:29 +01:00
Simon Frei
7e4e2f3720
lib/versioner: Improve error messages (fixes #7354) (#7357) 2021-02-12 20:30:51 +01:00
Jakob Borg
55afa625fc
cmd/syncthing: Add decrypt subcommand (#7332)
This adds the `syncthing decrypt` subcommand that is used to
(offline-)decrypt or just verify the contents of an encrypted folder.
2021-02-12 08:38:43 +01:00
Jakob Borg
4f20c900d0
cmd/syncthing: Refactor command line parsing (#7330) 2021-02-10 20:35:37 +01:00
Simon Frei
d842197931
lib/model: Disable tests involving scrypt with -short (fixes #7344) (#7346) 2021-02-08 17:13:28 +01:00
Simon Frei
c0f353c0e8
lib: Do not set ModifiedBy on meta only changes (#7345) 2021-02-08 15:30:39 +01:00
Simon Frei
11e9d575c8
lib/model: Refactor folder.scanSubdirs into smaller parts (#7321) 2021-02-08 08:40:57 +01:00