Commit Graph

1956 Commits

Author SHA1 Message Date
greatroar
6a9716e8a1 lib/model: Return index from deviceActivity.leastBusy
This way, we don't need a second loop over the Availabilities to remove
the selected item.
2021-11-26 12:07:43 +01:00
greatroar
8a1e54d58a lib/fs: Optimize Canonicalize
When pathSep is a constant, the compiler precomputes pathSep+pathSep and
".."+pathSep instead of emitting function calls to compute "//" and
"../". Benchmark results in lib/osutil:

name                old time/op    new time/op    delta
TraversesSymlink-8    8.86µs ± 3%    8.53µs ± 4%  -3.79%  (p=0.000 n=18+20)

name                old alloc/op   new alloc/op   delta
TraversesSymlink-8    1.06kB ± 0%    1.06kB ± 0%    ~     (all equal)

name                old allocs/op  new allocs/op  delta
TraversesSymlink-8      15.0 ± 0%      15.0 ± 0%    ~     (all equal)
2021-11-23 21:25:29 +01:00
greatroar
3e032c4da6 lib/fs: optimize Windows path checking/sanitizing
name                          old time/op    new time/op    delta
WindowsInvalidFilenameValid-8     875ns ± 1%     150ns ± 1%  -82.84%  (p=0.000 n=9+9)
WindowsInvalidFilenameNUL-8       276ns ± 4%     121ns ± 3%  -56.26%  (p=0.000 n=10+10)

name                          old alloc/op   new alloc/op   delta
WindowsInvalidFilenameValid-8     32.0B ± 0%     16.0B ± 0%  -50.00%  (p=0.000 n=10+10)
WindowsInvalidFilenameNUL-8       32.0B ± 0%     19.0B ± 0%  -40.62%  (p=0.000 n=10+10)

name                          old allocs/op  new allocs/op  delta
WindowsInvalidFilenameValid-8      2.00 ± 0%      1.00 ± 0%  -50.00%  (p=0.000 n=10+10)
WindowsInvalidFilenameNUL-8        2.00 ± 0%      2.00 ± 0%     ~     (all equal)
2021-11-23 21:25:29 +01:00
Jakob Borg
1754c93370
lib/config, lib/ignore: Write Windows line endings (fixes #7115) (#8052) 2021-11-22 09:38:24 +01:00
Jakob Borg
4b750b6dc3
all: Remove usage of deprecated io/ioutil (#7971)
As of Go 1.16 io/ioutil is deprecated. This replaces usage with the
corresponding functions in package os and package io.
2021-11-22 08:59:47 +01:00
greatroar
bf89bffb0b
lib/config: Decouple VerifyConfiguration from Committer (#7939)
... and remove 8/10 implementations, which were no-ops. This saves code
and time copying configurations.
2021-11-22 08:45:29 +01:00
Jakob Borg
e2288fe441
lib/relay: Send SNI when the address is a host name (fixes #8014) (#8015) 2021-11-22 08:31:03 +01:00
greatroar
8265dac127
lib/nat: Fix race condition in Mapping (#8042)
The locking protocol in nat.Mapping was racy:

* Mapping.addressMap RLock'd, but then returned a map shared between
  caller and Mapping, so the lock didn't do anything.

* Operations inside Service.{verifyExistingMappings,acquireNewMappings}
  would lock the map for every update, but that means callers to
  Mapping.ExternalAddresses can be looping over the map while the
  Service methods are concurrently modifying it. When the Go runtime
  detects that happening, it panics.

* Mapping.expires was read and updated without locking.

The Service methods now lock the map once and release the lock only when
done.

Also, subscribers no longer get the added and removed addresses, because
none of them were using the information. This was changed for a previous
attempt to retain the fine-grained locking and not reverted because it
simplifies the code.
2021-11-22 08:29:44 +01:00
André Colomb
100870e142
cmd/syncthing: Implement generate as a subcommand with optional API credential setting (fixes #8021) (#8043)
Accept a subcommand as an alternative to the --generate option.  It
accepts a custom config directory through either the --home or
--config options, using the default location if neither is given.

Add the options --gui-user and --gui-password to "generate", but not
the "serve --generate" option form.  If either is given, an existing
config will not abort the command, but rather load, modify and save it
with the new credentials.  The password can be read from standard
input by passing only a single dash as argument.

Config modification is skipped if the value matches what's already in
the config.

* cmd/syncthing: Utilize lib/locations package in generate().
Instead of manually joining paths with "magic" file names, get them
from the centralized locations helper lib.

* cmd/syncthing: Simplify logging for --generate option.
Visible change: No more timestamp prefixes.
2021-11-18 22:57:59 +01:00
Jakob Borg
12fb7f2a0a lib/model: Correct "reverting folder" log entry 2021-11-17 12:52:10 +01:00
Jakob Borg
f1bf4d899a lib/model: Correct handling of fakefs cache
We looked under one cache key, then stored under another...
2021-11-17 12:52:10 +01:00
Simon Frei
591e4d8af1
gui, lib: Fix tracking deleted locally-changed on encrypted (fixes #7715) (#7726) 2021-11-10 09:46:21 +01:00
André Colomb
dec6f80d2b
lib/config: Move the bcrypt password hashing to GUIConfiguration (#8028)
What hash is used to store the password should ideally be an
implementation detail, so that every user of the GUIConfiguration
object automatically agrees on how to handle it.  That is currently
distribututed over the confighandler.go and api_auth.go files, plus
tests.

Add the SetHasedPassword() / CompareHashedPassword() API to keep the
hashing method encapsulated.  Add a separate test for it and adjust
other users and tests.  Remove all deprecated imports of the bcrypt
package.
2021-11-08 13:32:04 +01:00
André Colomb
ec8a748514
lib/syncthing: Clean up / refactor LoadOrGenerateCertificate() utility function. (#8025)
LoadOrGenerateCertificate() takes two file path arguments, but then
uses the locations package to determine the actual path.  Fix that
with a minimally invasive change, by using the arguments instead.
Factor out GenerateCertificate().

The only caller of this function is cmd/syncthing, which passes the
same values, so this is technically a no-op.

* lib/tlsutil: Make storing generated certificate optional.  Avoid
  temporary cert and key files in tests, keep cert in memory.
2021-11-07 23:59:48 +01:00
greatroar
db15e52743
lib/api: http.Request.BasicAuth instead of custom code (#8039) 2021-11-06 12:38:08 +01:00
André Colomb
41bfb7a330
Normalize CLI options to always use two dashes. (#8037)
Consistently use double dashes and fix typos -conf, -data-dir and
-verify.

Applies also to tests running the syncthing binary for consistency.

* Fix mismatched option name --conf in cli subcommand.

According to the source code comments, the cli option flags should
mirror those from the serve subcommand where applicable.  That one is
actually called --config though.

* cli: Fix help text option placeholders.

The urfave/cli package uses the Value field of StringFlag to provide a
default value, not to name the placeholder.  That is instead done with
backticks around some part of the Usage field.

* cli: Add missing --data flag in subcommand help text.

The urfave/cli based option parsing uses a fake flags collection to
generate help texts matching the used global options.  But the --data
option was omitted from it, although it is definitely required when
using --config as well.  Note that it cannot just be ignored, as some
debug stuff actually uses the DB:

syncthing cli --data=/bar --config=/foo debug index dump
2021-11-04 08:42:55 +01:00
greatroar
807a6b1022
lib/model: Optimize jobQueue performance and memory use (#8023)
By truncating time.Time to an int64 nanosecond count, we lose the
ability to precisely order timestamps before 1678 or after 2262, but we
gain (linux/amd64, Go 1.17.1):

name                      old time/op    new time/op    delta
JobQueuePushPopDone10k-8    2.85ms ± 5%    2.29ms ± 2%  -19.80%  (p=0.000 n=20+18)
JobQueueBump-8              34.0µs ± 1%    29.8µs ± 1%  -12.35%  (p=0.000 n=19+19)

name                      old alloc/op   new alloc/op   delta
JobQueuePushPopDone10k-8    2.56MB ± 0%    1.76MB ± 0%  -31.31%  (p=0.000 n=18+13)

name                      old allocs/op  new allocs/op  delta
JobQueuePushPopDone10k-8      23.0 ± 0%      23.0 ± 0%     ~     (all equal)

Results for BenchmarkJobQueueBump are with the fixed version, which no
longer depends on b.N for the amount of work performed. rand.Rand.Intn
is cheap at ~10ns per iteration.
2021-10-29 20:20:46 +02:00
Tomasz Wilczyński
296cc1bca2 lib/model: Limit the number of default hashers on Android (ref #2220)
Like Windows and Mac, Android is also an interactive operating system.
On top of that, it usually runs on much slower hardware than the other
two. Because of that, it makes sense to limit the number of hashes used
by default there too.

Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
2021-10-29 10:06:52 +02:00
Simon Frei
951b058952 lib/model: Set mod. time after writing trailer in shortcut (ref #7992) 2021-10-24 16:50:30 +02:00
greatroar
8f8e8a9285 lib/protocol: Simplify codeToError, errorToCode
Also be explicit about the fact that ErrNoError is nil. That name isn't
used anywhere outside this file.
2021-10-22 18:40:40 +02:00
greatroar
46082f194c lib/protocol: Eliminate nativeModel on Unix 2021-10-22 18:40:40 +02:00
Simon Frei
517667c590
lib/model: Pull when a new connection is established (fixes #8012) (#8013) 2021-10-20 18:55:22 +02:00
greatroar
7c292cc812
lib/connections: Fix and optimize registry (#7996)
Registry.Get used a full sort to get the minimum of a list, and the sort
was broken because util.AddressUnspecifiedLess assumed it could find out
whether an address is IPv4 or IPv6 from its Network method. However,
net.(TCP|UDP)Addr.Network always returns "tcp"/"udp".
2021-10-06 10:52:51 +02:00
Simon Frei
c94b797f00
lib/protocol: Preserve sequence decrypting fileinfos (fixes #7994) (#7995) 2021-10-06 10:26:54 +02:00
Simon Frei
708a5c2070
lib/model: Write trailer when shortcutting on recv-enc (fixes #7991) (#7992) 2021-10-05 11:23:27 +02:00
Jakob Borg
92eaf52c21 lib/protocol: Test to lock down encryption key derivation
So that we don't inadvertently change how these things work.
2021-10-04 14:36:28 +02:00
tomasz1986
8d8f331a4a
lib/config: Always set mtime window to 2 on Android unless ext detected (ref #7963) (#7966)
The current detection is flawed, because it looks for a few specific
file systems like "msdos" or "fat" to set the mtime window, while in
reality Android seems to report names like "fuseblk", which can stand
for fat, ext4, or even f2fs.

At the moment, we set the mtime window only for a few known names used
for the fat filesystem. With this change, we take a safer approach of
always setting the time window unless we explicitly detect file systems
like ext2/ext3/ex4, which are known not to experience issues with moving
timestamps on Android.

Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
2021-09-27 21:29:51 +02:00
greatroar
198028d627
lib/rand: Optimizations (#7964)
rand.secureSource.Uint64 no longer allocates. rand.String uses a
strings.Builder. Benchmark results on linux/amd64:

name            old time/op    new time/op    delta
SecureSource-8    69.1ns ± 3%    51.7ns ± 3%   -25.21%  (p=0.000 n=20+10)
String-8          2.66µs ± 2%    1.95µs ± 1%   -26.61%  (p=0.000 n=10+10)

name            old alloc/op   new alloc/op   delta
SecureSource-8     8.00B ± 0%     0.00B       -100.00%  (p=0.000 n=20+10)
String-8            288B ± 0%       32B ± 0%   -88.89%  (p=0.000 n=10+10)

name            old allocs/op  new allocs/op  delta
SecureSource-8      1.00 ± 0%      0.00       -100.00%  (p=0.000 n=20+10)
String-8            33.0 ± 0%       1.0 ± 0%   -96.97%  (p=0.000 n=10+10)
2021-09-26 12:15:39 +02:00
Jakob Borg
73c5184518
build: Update all dependencies (#7968)
Also add a script step for how to do this, as frankly I forget each time
what the canonical way is.

`go run build.go update-deps`
2021-09-26 12:08:23 +02:00
greatroar
f96c211198
lib/db: Replace SipHash with hash/maphash (#7962) 2021-09-24 21:26:07 +02:00
greatroar
a4489dec30
lib/tlsutil: Allocate UnionedConnection in one go (#7912) 2021-09-21 08:40:34 +02:00
Simon Frei
721cd740d8
lib/model: Don't reset db while folder is running (fixes #7935) (#7936) 2021-09-11 17:14:47 +02:00
greatroar
de719ac409
lib/db: Inline sync.Once in releaser (#7938)
Having a pointer to a Once suggests that it is shared with other
objects, but it never is.
2021-09-10 09:58:17 +02:00
jtagcat
0ffa7f3f57
all: Clarify 'Cleaning data for folder' log message (#7937)
Instead of data (could be read as user data), use metadata.
2021-09-10 09:56:44 +02:00
greatroar
7fa141ea39
all: Unused args, retvals, assignments (#7926) 2021-09-08 00:11:16 +02:00
greatroar
13196ddd92 lib/relay/protocol: Merge two Sprintf calls 2021-09-06 15:30:56 +02:00
greatroar
eafb40460d lib/discover, lib/protocol: Buffer allocation 2021-09-06 15:30:56 +02:00
Simon Frei
4e2a9bb139
lib/model: Remove encryption pw on folder stop (#7925) 2021-09-03 09:54:47 +02:00
greatroar
3b2239357f
lib/model: Add missing error assignment + Remove unused argument (#7922) 2021-09-01 22:03:06 +02:00
Ross Smith II
7501bee430
lib/model: Typo in folder_recvonly.go (#7919) 2021-09-01 07:41:18 +02:00
greatroar
ed98039aa5
lib/fs: Optimize TempName + some cosmetic changes (#7911) 2021-08-29 10:47:53 +02:00
Jakob Borg
2816780b52
lib/api: Set "Secure" on session cookies served over HTTPS (ref #7399) (#7907)
So that it does not unnecessarily leak over clear text connections.
2021-08-27 17:56:54 +02:00
xjtdy888
48796a1b60
lib/ur: Fix panic build goroutines for failures (#7903) 2021-08-25 07:16:55 +02:00
Simon Frei
70a840d3d5
lib/connections: Remove future go build constraints on quic (#7898) 2021-08-19 21:05:28 +02:00
Jakob Borg
d47745a86b
all: Update build constraints to Go 1.17 style (#7894) 2021-08-17 10:10:41 +02:00
Simon Frei
0fe72e6fc5
lib/model: Schedule pull after revert on recv-enc (#7892) 2021-08-17 09:24:10 +02:00
Simon Frei
c025e76f30
lib/model: Do not wait for scan when setting ignores (fixes #7893) (#7891) 2021-08-17 09:23:33 +02:00
Simon Frei
e1bf1e672e
lib/ur: Fix panic getting goroutines for failures (ref #7785) (#7890) 2021-08-16 22:47:05 +02:00
Simon Frei
db302b15ea
lib/syncthing: Set system timezone on android (#7878) 2021-08-09 09:27:14 +02:00
Simon Frei
952f3ffb0c
lib/db/backend: Prevent panic in errors.As (#7873) 2021-08-05 11:25:20 +01:00
Audrius Butkevicius
fe77fac23f
Update quic to 0.22.0 to support RFC9000, enable batch reads (#7862)
* Update quic to 0.22.0 to support RFC9000, enable batch reads
* Remove wrappers that are not needed anymore
2021-08-05 05:44:22 +02:00
Simon Frei
e61091d240
lib/stun: Prevent nil deref when naming service (#7872) 2021-08-05 00:04:22 +01:00
Simon Frei
50aacdf1f0
lib/protocol: Ensure correct blocksize on enc. fileinfo (ref #7861) (#7870) 2021-08-04 23:12:01 +02:00
Simon Frei
e56e8b7aa1
lib/model: Don't consider hashes pulling on recv-enc (#7869) 2021-08-04 11:10:10 +02:00
tomasz1986
5a1f6cb813
lib/fs: Improve case conflict error message (fixes #7827) (#7829) 2021-08-01 22:44:49 +02:00
greatroar
37d0ba1660
lib/watchaggregator: Replace counter map by two integers (#7856) 2021-08-01 12:30:20 +02:00
Jonathan
0836439256
lib/model: Fix config deadlock when deleting a paused folder (#7854) 2021-07-30 14:41:00 +02:00
Simon Frei
67b18569cf
all: Send deadlocks as failures, crash only as a last resort (#7785) 2021-07-27 21:27:52 +02:00
Simon Frei
dc0dd09e93
lib/model: Don't try to delete deleted item on revert (#7843) 2021-07-23 14:26:20 +02:00
Simon Frei
eeb7091180
lib/model: Missing fmut-lock on encryption failures (#7841) 2021-07-22 11:15:25 +02:00
greatroar
dc38e6ae88
lib/relay/client: Stricter typing and remove unused code (#7819) 2021-07-21 09:49:09 +02:00
greatroar
1e21042138
lib/connections: switch statement to get the QUIC network (#7816) 2021-07-10 13:53:51 +02:00
Chih-Hsuan Yen
11c57b9097
lib/connections: Resolve IPv6 for quic6:// peers (fixes #7809) (#7810)
Before this patch, IPv4-compatible addresses (::ffff:aaa.bbb.ccc.ddd)
may be used if a quic6://some.domain:port is specified and both IPv4 and
IPv6 addresses exist for that domain name.
2021-07-05 13:19:56 +02:00
Simon Frei
1921533c4c
lib/connections: Fully dial resumed devices (#7798) 2021-07-03 18:26:55 +02:00
Simon Frei
89e762fd6e
lib/model: Prevent folder-type change from/to encrypted (fixes #7704) (#7796) 2021-07-03 13:47:04 +02:00
Simon Frei
a63d3ee625
lib/model: Scan removed dirs when reverting recv-enc (fixes #7706) (#7797) 2021-07-03 13:46:24 +02:00
greatroar
bd363fe0b7
lib/protocol: Write uncompressible messages uncompressed (#7790) 2021-06-27 17:59:30 +02:00
Simon Frei
445a82f120
lib/model: Compare all items with global on scan (fixes #7740) (#7791) 2021-06-27 08:48:54 +02:00
Simon Frei
69ce121267
lib/db: Missing event-logger in write-transaction (#7793) 2021-06-27 08:43:49 +02:00
Simon Frei
08e3cd1cce
lib/fs: Set expiry after DirNames in case-fs (#7794) 2021-06-27 08:30:02 +02:00
Simon Frei
c78fa42f31
lib/connections: Dial devices in parallel (#7783) 2021-06-25 11:38:04 +02:00
Jakob Borg
993a3ebe73
lib/api: Always include usage reporting data in support bundle (#7786) 2021-06-24 22:00:14 +04:00
Simon Frei
400d62c1e6
lib/connections: Missed map init in nextDialAt (ref #7753) (#7778) 2021-06-17 21:13:57 +04:00
Simon Frei
857caf3637
lib/connections: Trigger dialer when connection gets closed (#7753) 2021-06-17 13:57:44 +02:00
Simon Frei
aeca1fb575
lib/db: Check if sequences change when repairing metadata (#7770) 2021-06-17 13:53:39 +02:00
Simon Frei
ac2988a485
gui, lib: Handle pw adding remote encrypted folder (fixes #7705) (#7772) 2021-06-17 13:53:02 +02:00
Simon Frei
23a0e18292
lib/db: Fix accounting bug when dropping indexes (#7774) 2021-06-17 10:15:11 +02:00
Simon Frei
aa6c55dec1
lib/model: Remove bogus failureevent when restarting folder (#7773) 2021-06-17 08:57:24 +02:00
Simon Frei
1e7a3997e3
lib/db, lib/model: Improve error handling on pending items (#7754) 2021-06-09 13:35:17 +02:00
Anur
e7f8538e4d
lib/fs: Add bitmasks for Darwin to handle change to empty files (fixes #7731) (#7756) 2021-06-09 12:57:06 +02:00
Simon Frei
1a22689328
lib/db: Add failure reports to failures iterating over hashes (#7755) 2021-06-07 23:10:35 +02:00
Jakob Borg
ce65aea0ab
lib/db: Use a more concurrent GC (fixes #7722) (#7750)
This changes the GC mechanism so that the first pass (which reads all
FileInfos to populate bloom filters with block & version hashes) can
happen concurrently with normal database operations.

The big gcMut still exists, and we grab it temporarily to block all
other modifications while we set up the bloom filters. We then release
the lock and let other things happen, with those other things also
updating the bloom filters as required. Once the first phase is done we
again grab the gcMut, knowing that we are the sole modifier of the
database, and do the cleanup.

I also removed the final compaction step.
2021-06-07 10:52:06 +02:00
André Colomb
45edad867c
all: Allow dismissing pending devices / folders without ignoring (fixes #7700) (#7712) 2021-06-07 10:29:24 +02:00
André Colomb
ea0a408849
gui: Modal dialog for listeners and discovery status (#7539) 2021-06-07 09:08:44 +02:00
Simon Frei
18592af993
lib/model: Fix wrongly hardcoded arguments in test helper (#7749) 2021-06-05 17:01:23 +02:00
Simon Frei
b1e0e7b923
lib/model: Fix indexhandling for new folders paused on remote (#7747) 2021-06-05 16:27:15 +02:00
Simon Frei
6494a9332d
lib/model: Fix test introduced in #7714 failing due to #7689 (#7745) 2021-06-04 15:32:47 +02:00
Simon Frei
41baccb85d
lib/model: Fix passwords on receive-enc needing token (ref #7518) (#7739) 2021-06-03 15:39:49 +02:00
Simon Frei
52eb7392c4
lib/api, lib/config: Apply defaults before deserializing json (#7690) 2021-06-03 15:09:35 +02:00
Simon Frei
855c53ad02
lib/model: Fix reverting when version has only our own ID (fixes #7708) (#7714) 2021-06-03 15:08:56 +02:00
Simon Frei
004eded398
lib/model: Don't share with introduced device if encrypted (fixes #7724) (#7734) 2021-06-03 15:02:57 +02:00
Simon Frei
df48276300
lib/model: Ensure indexes are only received after checking IDs (ref #7649) (#7689) 2021-06-03 14:58:50 +02:00
greatroar
95c9561e97
lib/db: Clean up Timer and wait for logging before return in GC (#7720) 2021-05-31 09:50:21 +02:00
Simon Frei
fcb19518c7
build, lib/model: Add flag to run tests without -short and fix failure (#7716) 2021-05-28 22:20:18 +02:00
Simon Frei
22e09334ec
lib/model: Fix incoming request on receive-enc (fixes #7699) (#7702) 2021-05-22 21:38:49 +02:00
Simon Frei
58592e3ef1
lib/db: Add logging for GC (#7707) 2021-05-22 21:36:43 +02:00
Simon Frei
0126188ba7
lib/config: Set DisableTempIndexes to true on receive-encrypted (#7701) 2021-05-20 22:33:23 +02:00
Simon Frei
5bdb6798a9
all: Regenerate proto (#7696) 2021-05-19 13:30:20 +02:00
greatroar
0e52ce830a
lib/fs: Fix UnicodeLowercaseNormalized on lowercase NFD (#7692)
Co-authored-by: greatroar <@>
2021-05-17 20:43:07 +02:00
Jakob Borg
97437cad64
lib/fs: Ignore normalization differences in case insensitive lookup (fixes #7677) (#7678) 2021-05-17 12:35:03 +02:00