Commit Graph

2104 Commits

Author SHA1 Message Date
Simon Frei
50aacdf1f0
lib/protocol: Ensure correct blocksize on enc. fileinfo (ref #7861) (#7870) 2021-08-04 23:12:01 +02:00
Simon Frei
e56e8b7aa1
lib/model: Don't consider hashes pulling on recv-enc (#7869) 2021-08-04 11:10:10 +02:00
tomasz1986
5a1f6cb813
lib/fs: Improve case conflict error message (fixes #7827) (#7829) 2021-08-01 22:44:49 +02:00
greatroar
37d0ba1660
lib/watchaggregator: Replace counter map by two integers (#7856) 2021-08-01 12:30:20 +02:00
Jonathan
0836439256
lib/model: Fix config deadlock when deleting a paused folder (#7854) 2021-07-30 14:41:00 +02:00
Simon Frei
67b18569cf
all: Send deadlocks as failures, crash only as a last resort (#7785) 2021-07-27 21:27:52 +02:00
Simon Frei
dc0dd09e93
lib/model: Don't try to delete deleted item on revert (#7843) 2021-07-23 14:26:20 +02:00
Simon Frei
eeb7091180
lib/model: Missing fmut-lock on encryption failures (#7841) 2021-07-22 11:15:25 +02:00
greatroar
dc38e6ae88
lib/relay/client: Stricter typing and remove unused code (#7819) 2021-07-21 09:49:09 +02:00
greatroar
1e21042138
lib/connections: switch statement to get the QUIC network (#7816) 2021-07-10 13:53:51 +02:00
Chih-Hsuan Yen
11c57b9097
lib/connections: Resolve IPv6 for quic6:// peers (fixes #7809) (#7810)
Before this patch, IPv4-compatible addresses (::ffff:aaa.bbb.ccc.ddd)
may be used if a quic6://some.domain:port is specified and both IPv4 and
IPv6 addresses exist for that domain name.
2021-07-05 13:19:56 +02:00
Simon Frei
1921533c4c
lib/connections: Fully dial resumed devices (#7798) 2021-07-03 18:26:55 +02:00
Simon Frei
89e762fd6e
lib/model: Prevent folder-type change from/to encrypted (fixes #7704) (#7796) 2021-07-03 13:47:04 +02:00
Simon Frei
a63d3ee625
lib/model: Scan removed dirs when reverting recv-enc (fixes #7706) (#7797) 2021-07-03 13:46:24 +02:00
greatroar
bd363fe0b7
lib/protocol: Write uncompressible messages uncompressed (#7790) 2021-06-27 17:59:30 +02:00
Simon Frei
445a82f120
lib/model: Compare all items with global on scan (fixes #7740) (#7791) 2021-06-27 08:48:54 +02:00
Simon Frei
69ce121267
lib/db: Missing event-logger in write-transaction (#7793) 2021-06-27 08:43:49 +02:00
Simon Frei
08e3cd1cce
lib/fs: Set expiry after DirNames in case-fs (#7794) 2021-06-27 08:30:02 +02:00
Simon Frei
c78fa42f31
lib/connections: Dial devices in parallel (#7783) 2021-06-25 11:38:04 +02:00
Jakob Borg
993a3ebe73
lib/api: Always include usage reporting data in support bundle (#7786) 2021-06-24 22:00:14 +04:00
Simon Frei
400d62c1e6
lib/connections: Missed map init in nextDialAt (ref #7753) (#7778) 2021-06-17 21:13:57 +04:00
Simon Frei
857caf3637
lib/connections: Trigger dialer when connection gets closed (#7753) 2021-06-17 13:57:44 +02:00
Simon Frei
aeca1fb575
lib/db: Check if sequences change when repairing metadata (#7770) 2021-06-17 13:53:39 +02:00
Simon Frei
ac2988a485
gui, lib: Handle pw adding remote encrypted folder (fixes #7705) (#7772) 2021-06-17 13:53:02 +02:00
Simon Frei
23a0e18292
lib/db: Fix accounting bug when dropping indexes (#7774) 2021-06-17 10:15:11 +02:00
Simon Frei
aa6c55dec1
lib/model: Remove bogus failureevent when restarting folder (#7773) 2021-06-17 08:57:24 +02:00
Simon Frei
1e7a3997e3
lib/db, lib/model: Improve error handling on pending items (#7754) 2021-06-09 13:35:17 +02:00
Anur
e7f8538e4d
lib/fs: Add bitmasks for Darwin to handle change to empty files (fixes #7731) (#7756) 2021-06-09 12:57:06 +02:00
Simon Frei
1a22689328
lib/db: Add failure reports to failures iterating over hashes (#7755) 2021-06-07 23:10:35 +02:00
Jakob Borg
ce65aea0ab
lib/db: Use a more concurrent GC (fixes #7722) (#7750)
This changes the GC mechanism so that the first pass (which reads all
FileInfos to populate bloom filters with block & version hashes) can
happen concurrently with normal database operations.

The big gcMut still exists, and we grab it temporarily to block all
other modifications while we set up the bloom filters. We then release
the lock and let other things happen, with those other things also
updating the bloom filters as required. Once the first phase is done we
again grab the gcMut, knowing that we are the sole modifier of the
database, and do the cleanup.

I also removed the final compaction step.
2021-06-07 10:52:06 +02:00
André Colomb
45edad867c
all: Allow dismissing pending devices / folders without ignoring (fixes #7700) (#7712) 2021-06-07 10:29:24 +02:00
André Colomb
ea0a408849
gui: Modal dialog for listeners and discovery status (#7539) 2021-06-07 09:08:44 +02:00
Simon Frei
18592af993
lib/model: Fix wrongly hardcoded arguments in test helper (#7749) 2021-06-05 17:01:23 +02:00
Simon Frei
b1e0e7b923
lib/model: Fix indexhandling for new folders paused on remote (#7747) 2021-06-05 16:27:15 +02:00
Simon Frei
6494a9332d
lib/model: Fix test introduced in #7714 failing due to #7689 (#7745) 2021-06-04 15:32:47 +02:00
Simon Frei
41baccb85d
lib/model: Fix passwords on receive-enc needing token (ref #7518) (#7739) 2021-06-03 15:39:49 +02:00
Simon Frei
52eb7392c4
lib/api, lib/config: Apply defaults before deserializing json (#7690) 2021-06-03 15:09:35 +02:00
Simon Frei
855c53ad02
lib/model: Fix reverting when version has only our own ID (fixes #7708) (#7714) 2021-06-03 15:08:56 +02:00
Simon Frei
004eded398
lib/model: Don't share with introduced device if encrypted (fixes #7724) (#7734) 2021-06-03 15:02:57 +02:00
Simon Frei
df48276300
lib/model: Ensure indexes are only received after checking IDs (ref #7649) (#7689) 2021-06-03 14:58:50 +02:00
greatroar
95c9561e97
lib/db: Clean up Timer and wait for logging before return in GC (#7720) 2021-05-31 09:50:21 +02:00
Simon Frei
fcb19518c7
build, lib/model: Add flag to run tests without -short and fix failure (#7716) 2021-05-28 22:20:18 +02:00
Simon Frei
22e09334ec
lib/model: Fix incoming request on receive-enc (fixes #7699) (#7702) 2021-05-22 21:38:49 +02:00
Simon Frei
58592e3ef1
lib/db: Add logging for GC (#7707) 2021-05-22 21:36:43 +02:00
Simon Frei
0126188ba7
lib/config: Set DisableTempIndexes to true on receive-encrypted (#7701) 2021-05-20 22:33:23 +02:00
Simon Frei
5bdb6798a9
all: Regenerate proto (#7696) 2021-05-19 13:30:20 +02:00
greatroar
0e52ce830a
lib/fs: Fix UnicodeLowercaseNormalized on lowercase NFD (#7692)
Co-authored-by: greatroar <@>
2021-05-17 20:43:07 +02:00
Jakob Borg
97437cad64
lib/fs: Ignore normalization differences in case insensitive lookup (fixes #7677) (#7678) 2021-05-17 12:35:03 +02:00
Simon Frei
5b90a98650
lib/model: Fix addFakeConn and other test improvements (#7684) 2021-05-16 17:23:27 +02:00
Simon Frei
93a02c677e
lib/scanner: Do not drop all not-exist-errors and debug logging (#7675) 2021-05-15 11:51:35 +02:00
Simon Frei
0d054f9b64
lib/model: Don't use empty folder cfg for index sender (fixes #7649) (#7671) 2021-05-15 11:13:39 +02:00
Audrius Butkevicius
1107f6eb5f
lib/connections: Reduce default quic redial interval (fixes #7471) (#7672)
* lib/connections: Reduce default quic redial interval (fixes #7471)

* Update quic_dial.go
2021-05-14 14:26:02 +01:00
Simon Frei
c494ced21f
lib/connections: Actually remove listenerSupervisor (ref #7644) (#7663) 2021-05-11 14:35:13 +02:00
Audrius Butkevicius
aca1b45e93
lib/connections: Update pfilter to pick up bugfix/oob stuff, support OOB connections (fixes #7636) (#7654) 2021-05-11 07:59:56 +02:00
Simon Frei
5cb2a10138
lib/model: Improve encryption cluster-config errors (#7658) 2021-05-11 07:55:44 +02:00
Audrius Butkevicius
411796606c
lib/connections: Correct service termination order (#7657) 2021-05-10 22:29:27 +02:00
Simon Frei
1a9b54c9fa
lib/connections: Separate listener supervisors and lower backoff time (#7644) 2021-05-10 22:26:51 +02:00
Simon Frei
c7f4f15272
lib/relay, lib/svcutil: Improve service logging (fixes #7580) (#7647) 2021-05-10 22:26:25 +02:00
Simon Frei
713527facf
all: Refactor relay invitations (#7646) 2021-05-10 22:25:43 +02:00
Simon Frei
6e662dc9fc
lib/suture: Use ServeBackground to start main supervisor (#7626) 2021-05-10 16:50:45 +02:00
Audrius Butkevicius
eb178caf3a
lib/connections: Add connection benchmarks, allow binding to port zero addresses (#7648)
* Add connbench

* Refactor port fixup

* More cleanup

* touch for build

Co-authored-by: Jakob Borg <jakob@kastelo.net>
2021-05-10 15:44:47 +01:00
Simon Frei
6157c766de
lib/connections: Correct comments on quic wrapper type (#7652) 2021-05-09 19:15:10 +01:00
Audrius Butkevicius
745cd4744a
lib/connections: Revert add more methods to the quic conn wrapper (#7651)
This reverts commit faf15b4567.
2021-05-09 19:43:16 +02:00
Simon Frei
faf15b4567
lib/connections: Add more methods to the quic conn wrapper (#7643) 2021-05-09 12:45:08 +01:00
greatroar
3746c899b7
build: List go:generate tools in tools.go (#7599) 2021-05-08 12:52:06 +02:00
Audrius Butkevicius
87a0eecc31
lib/fs, lib/api, lib/model: Expose mtime remappings as part of /db/file (#7624)
* lib/fs, lib/api, lib/model: Expose mtime remappings as part of /db/file

* Fix wrong error returned by CLI

* Gofmt

* Better names

* Review comments

* Review comments
2021-05-03 11:28:25 +01:00
Simon Frei
75b58eb480
lib/fs: Watch attrib. changes on inotify for mod. time (#7623)
* lib/fs: Watch attrib. changes on inotify for mod. time

* fix bsds (real) and darwin (test only)
2021-05-02 16:45:44 +02:00
Simon Frei
dd39556759
lib: Revert needing invalid files (fixes #7608, ref #7476) (#7609) 2021-04-29 22:01:46 +02:00
Simon Frei
1f30383866
lib/model: Remove path from enc errors and report only once (#7610) 2021-04-29 19:21:07 +02:00
Gahl Saraf
66662cd678
Trigger connection loop on config device addition (fixes #7600) (#7604)
* Trigger connection loop on config device addition (fixes #7600)

* Also check for device address equality

* Move EqualStrings from api_test to utils, and use in connections/service.go

* Make sure CommitConfiguration cannot block due on the deviceAddressesChanged channel

* Update lib/connections/service.go

Co-authored-by: Jakob Borg <jakob@kastelo.net>
2021-04-26 21:13:59 +01:00
Simon Frei
8734fa65fc
lib/model: Handle invalid needed items on send-only (ref #7476) (#7596) 2021-04-26 15:36:51 +02:00
Jakob Borg
74823e81e9
all: Deprecate TLS 1.2 on sync connections (fixes #7594) (#7598)
This makes us use TLS 1.3+ on sync connections by default. A new option
`insecureAllowOldTLSVersions` exists to allow communication with TLS
1.2-only clients (roughly Syncthing 1.2.2 and older). Even with that
option set you get a slightly simplified setup, with the cipher suite
order fixed instead of auto detected.
2021-04-26 10:04:35 +02:00
Jakob Borg
e4c995a321
lib/model: Don't remove non-empty .stfolder (#7578) 2021-04-15 18:13:35 +02:00
André Colomb
130d14cec9
api: Log API authorization failures. (#7575) 2021-04-15 07:33:02 +02:00
Simon Frei
15d2dc3a4f
lib/connections: Add SyscallConn() to quic conn (fixes #7551) (#7570) 2021-04-13 12:59:58 +01:00
André Colomb
f6df1a760d
lib/api: Log the remote address on login attempts (#7560)
This enables usage of the audit log to e.g. automatically block remote
addresses from connecting after repeated login failures.
2021-04-13 10:14:44 +02:00
Simon Frei
f71fcd440a
all: Remove untrusted feature flag (fixes #109) (#7567)
No longer hide the web UI controls for the new untrusted/encrypted
device feature. Testing hasn't been very widespread, but there has been
some and quite a few bugs have been caught and fixed. I believe its time
to not hide it anymore, and cautiously recommend usage. E.g. mention
that the feature hasn't been widely used yet and anyone using it is an
early adopter, but drop the bit about not using it with production data.
We can maybe stress the need for backups in general and especially
using this.
2021-04-13 10:12:56 +02:00
Simon Frei
1a00ea7c6e
lib: Prevent using protocol method with native path (fixes #7557) (#7563) 2021-04-11 15:29:43 +02:00
Simon Frei
ec0a66c75b
lib/db, lib/model: Refactor removing expired pending folders (#7537) 2021-04-11 15:24:08 +02:00
Jakob Borg
8e38ecdeb2
Merge pull request from GHSA-x462-89pf-6r5h 2021-04-06 08:00:00 +02:00
Simon Frei
f30f9c50f8
lib/db: Handle indirection error repairing sequences (fixes #7026) (#7525) 2021-04-05 10:24:16 +02:00
Simon Frei
0dcd9794d4
lib/protocol: Deterministic encrypted version (fixes #7533) (#7538) 2021-03-31 08:59:15 +02:00
André Colomb
fb2d85b9d5
lib/db: Fix comparison of pending folder timestamps (fixes #7532) (#7535) 2021-03-30 21:24:53 +02:00
Simon Frei
7f0d4f6ba8
lib/db: Don't panic debugging an inexistent file (#7534) 2021-03-30 20:06:01 +02:00
Simon Frei
0d7a77ba85
lib/model: Reset conn when enc token is missing (fixes #7198) (#7518) 2021-03-23 10:38:40 +01:00
Simon Frei
924b96856f
lib: Handle adding enc folders on an existing conn (fixes #7509) (#7510) 2021-03-22 21:50:19 +01:00
Simon Frei
6b25eb2e79
lib/ur: Prevent panic when blocksResult is nil (ref #7495) (#7515) 2021-03-22 15:07:41 +01:00
Simon Frei
bc08a951f1
lib/model: Encrypted fileinfo trailer needs to be in wire format (#7505) 2021-03-21 10:34:08 +01:00
Lars Lehtonen
a87c5515bd
lib/model: Error check in test (#7508) 2021-03-21 10:32:17 +01:00
Jakob Borg
ebcd22b02b
lib/osutil: Fix raising max FDs on macOS (#7507)
There was a logic mistake, so the limit in question wasn't used. On my
macOS this doesn't seem to matter, the hard limit returned is 2^63-1 and
setting the soft limit to that works. However I'm assuming that's not
the case for older macOSes since it was so nicely documented, so we
should still have this working. (10240 FDs should be enough for
anybody.)
2021-03-20 16:32:36 +01:00
Audrius Butkevicius
4b02b7e6f1
lib/connections: Silence "connected to myself" messages. (#7500) 2021-03-17 23:53:20 +01:00
Jakob Borg
fdd823d2cb lib/osutil: Remove unused code 2021-03-17 23:18:07 +01:00
Jakob Borg
8ef504f745
all: Simplify some method calls (#7499)
strings.Replace(a, b, c, -1) -> strings.ReplaceAll(a, b, c)

(Go 1.12) and who knows what was up with that dialQueue.Sort() thing.
2021-03-17 23:12:26 +01:00
deepsource-autofix[bot]
6c573a5762
Remove unnecessary guard around delete (#7496)
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>
2021-03-17 22:23:36 +01:00
Jakob Borg
3ac858b150
all: Remove miscellaneous vestigial code (#7495) 2021-03-17 22:23:12 +01:00
Jakob Borg
f4372710bf
all: Remove crypto/md5 (#7493)
This is a mostly pointless change to make security scanners and static
analysis tools happy, as they all hate seeing md5. None of our md5 uses
were security relevant, but still. Only visible effect of this change is
that our temp file names for very long file names become slightly longer
than they were previously...
2021-03-17 22:22:49 +01:00
Jakob Borg
f39477bbd5
lib/api: Missing error handling in API delete-device (#7494) 2021-03-17 22:08:44 +01:00
Jakob Borg
6e5514419d
lib/db: Fix some omitted error checks, unused variable (#7489) 2021-03-17 21:41:07 +01:00
Jakob Borg
81484699db
lib/model: Actually break puller loop on context cancel (#7492)
Current break does nothing (breaks the select).
2021-03-17 21:34:52 +01:00
deepsource-autofix[bot]
0930bccf88
cmd/ursrv, lib/scanner: Remove unnecessary slicing of slices (#7491)
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>
2021-03-17 21:04:36 +01:00
Jakob Borg
e321bd3941
lib/*/auto: Add noassets files (#7490)
This adds a couple of dummy asset files protected by the "noassets"
build tag. The purpose is that it should be possible for, for example,
CI tools and static analysis things to compile and analyze the source
tree without our custom asset generation step. Also makes `go test -tags
noassets ./...` work without building assets first.
2021-03-17 21:03:35 +01:00
Simon Frei
4b02937862
lib/model: Add missing lock on availability func (#7487) 2021-03-17 20:46:13 +01:00
Simon Frei
3f2b584c4e
lib/model: Don't use ignore patterns for recv-enc folders (fixes #7469) (#7472) 2021-03-16 15:04:11 +01:00
Simon Frei
d2d4fcc1df
lib/protocol: Improve messages when an error occurs receiving (ref #7466) (#7470) 2021-03-15 19:14:09 +01:00
Simon Frei
273ee09925
lib/db, lib/model: Allow needing invalid files (fixes #7474) (#7476) 2021-03-15 07:58:01 +01:00
Simon Frei
f80ee472c2
lib/protocol: Set invalid flag on encrypted file infos (fixes #7466) (#7467) 2021-03-13 16:57:36 +01:00
Simon Frei
97a8777d03
lib/fs: Check both old and new path when renaming (fixes #7426) (#7463) 2021-03-12 21:15:50 +01:00
Simon Frei
8a4c00d82e
lib/model: Send failure report on CC encryption check error (#7460) 2021-03-12 12:21:54 +01:00
Simon Frei
31f859e909
lib/model: Return correct error in puller-iteration (ref #7424) (#7461) 2021-03-12 12:21:28 +01:00
Jakob Borg
4d979a1ce9
all: Truncate some timestamps (fixes #7457) (#7459)
This truncates times meant for API consumption to second precision,
where fractions won't typically matter or add any value. Exception to
this is timestamps on logs and events, and of course I'm not touching
things like file metadata.

I'm not 100% certain this is an exhaustive change, but it's the things I
found by grepping and following the breadcrumbs from lib/api...

I also considered general-but-ugly solutions, like having the API
serializer itself do reflection magic or even regexps on returned
objects, but decided against it because aurgh...
2021-03-12 10:35:10 +01:00
Simon Frei
4465cdf8bc
lib/api: Fix body of renamed config/restart-required endpoint (ref #7402) (#7453) 2021-03-11 15:54:05 +01:00
Simon Frei
3938b61c3f
lib/fs: Expose fs option on interface (fixes #7385, ref #7381) (#7389) 2021-03-11 15:23:56 +01:00
Jakob Borg
cdef503db6
all: Make config.Wrapper an actual suture.Service (fixes #7451) (#7452) 2021-03-11 14:51:00 +01:00
Jakob Borg
df08984a58
lib/api: Sanitize names used in certificates (fixes #7434) (#7435) 2021-03-11 13:15:03 +01:00
Simon Frei
9a001051d6
cmd/ursrv, lib/ur: Collect and present encryption usage (#7448) 2021-03-10 22:26:56 +00:00
greatroar
c00520281b
lib/protocol: Optimize FileKey (#7440) 2021-03-07 18:44:21 +01:00
Simon Frei
310fba4c12
lib: Return error from db.FileSet.Snapshot (fixes #7419, ref #5907) (#7424) 2021-03-07 13:43:22 +01:00
Simon Frei
767e1c6f58
lib/connections: Expose SetReadBuffer on conn passed to quic (ref #7417) (#7432) 2021-03-05 08:06:37 +01:00
Simon Frei
3d91f7c975
lib: Use counterfeiter to mock interfaces in tests (#7375) 2021-03-03 08:53:50 +01:00
greatroar
56b5352f64
all: Use crypt/rand through its buffered version, but not in benchmarks (#7420) 2021-03-02 19:17:20 +01:00
Jakob Borg
55d5e03639
lib/db: Remove Badger experiment (#7413) 2021-03-01 09:16:08 +01:00
greatroar
ffcaffa32f
lib/protocol: Optimize encrypted filename handling + make it more strict (#7408) 2021-02-27 08:57:12 +01:00
Simon Frei
0ffd80f380
lib/protocol: Alwasy return buffers to the pool (#7409) 2021-02-27 08:55:51 +01:00
wouter bolsterlee
25151b14e7
lib/api: Treat *.localhost as valid localhost addresses (#7412) (ref #4815)
This loosens the ‘is this localhost?’ check to include *.localhost host
names.

This allows for clearer (hence better) names to be used in browsers,
e.g. when accessing a remote syncthing instance ‘foo’ using a ssh port
forward, one can use foo.localhost to remind oneself which one is which.
💡 Without these changes, Syncthing shows a ‘Host check error’ when
pointing a browser at http://foo.localhost/, and with these changes, the
interface loads as usual.

The .localhost top level domain is a reserved top-level domain (RFC 2606):

> The ".localhost" TLD has traditionally been statically defined in
> host DNS implementations as having an A record pointing to the
> loop back IP address and is reserved for such use.  Any other use
> would conflict with widely deployed code which assumes this use.
> – https://tools.ietf.org/html/rfc2606

As Wikipedia puts it:

> This allows the use of these names for either documentation purposes
or in local testing scenarios. – https://en.wikipedia.org/wiki/.localhost

On Linux systems, systemd-resolved resolves *.localhost, on purpose:
https://www.freedesktop.org/software/systemd/man/systemd-resolved.service.html

See also #4815, #4816.
2021-02-27 08:52:49 +01:00
Simon Frei
fff8805ff6
all: Fix versioning path handling (#7407) 2021-02-26 12:04:05 +01:00
Simon Frei
0bf9645f2f
lib/api: Rename config insync endpoint to restart-required (#7402) 2021-02-25 09:29:44 +01:00
tomasz1986
1ed0116147
all: Fix Microsoft documentation links in code comments (#7387) 2021-02-20 14:56:45 +01:00
Simon Frei
aa1c274231
lib/fs: Consider options in case-fs caching (fixes #7371) (#7381) 2021-02-19 11:06:25 +01:00
Simon Frei
78c2844e3f
lib/scanner: Pass on errors while hashing (#7380) 2021-02-19 08:51:39 +01:00
Jakob Borg
327604719a
lib/connections: Allow QUIC with Go 1.16 (#7372) 2021-02-17 11:09:16 +01:00
Audrius Butkevicius
fb078068b4
cmd/syncthing: Add cli as a subcommand (fixes #6566, fixes #4719) (#7364)
* cmd/syncthing: Add cli as a subcommand (fixes #6566, fixes #4719)

* Hijack help

* Add comment

* Revert go.mod/go.sum
2021-02-15 18:50:53 +01:00
Simon Frei
80fc238bec
all: Automatic/disabled folder-config when receive-encrypted (#7327) 2021-02-12 22:51:29 +01:00
Simon Frei
7e4e2f3720
lib/versioner: Improve error messages (fixes #7354) (#7357) 2021-02-12 20:30:51 +01:00
Jakob Borg
55afa625fc
cmd/syncthing: Add decrypt subcommand (#7332)
This adds the `syncthing decrypt` subcommand that is used to
(offline-)decrypt or just verify the contents of an encrypted folder.
2021-02-12 08:38:43 +01:00
Jakob Borg
4f20c900d0
cmd/syncthing: Refactor command line parsing (#7330) 2021-02-10 20:35:37 +01:00
Simon Frei
d842197931
lib/model: Disable tests involving scrypt with -short (fixes #7344) (#7346) 2021-02-08 17:13:28 +01:00
Simon Frei
c0f353c0e8
lib: Do not set ModifiedBy on meta only changes (#7345) 2021-02-08 15:30:39 +01:00
Simon Frei
11e9d575c8
lib/model: Refactor folder.scanSubdirs into smaller parts (#7321) 2021-02-08 08:40:57 +01:00
Simon Frei
46bbc78e82
lib/db: Fix and improve removing entries from global (ref #6501) (#7336) 2021-02-08 08:38:41 +01:00
Jakob Borg
6db8dc33f2
lib/model: Correctly verify short read blocks (fixes #7333) (#7334)
An untrusted device will receive padded info for small blocks, and hence
sometimes request a larger block than actually exists on disk.
Previously we let this pass because we didn't have a hash to compare to
in that case and we ignored the EOF error based on that.

Now the untrusted device does pass an encrypted hash that we decrypt and
verify. This means we can't check for len(hash)==0 any more, but on the
other hand we do have a valid hash we can apply to the data we actually
read. If it matches then we don't need to worry about the read
supposedly being a bit short.
2021-02-05 16:07:21 +01:00
Jakob Borg
194501c958
lib/api: Give the config changes some more time? (#7335) 2021-02-05 15:33:37 +01:00
Simon Frei
27a34609a1
all: Failure reporting fixes (#7331) 2021-02-05 11:21:14 +01:00
Simon Frei
ffc14a77c6
all: Add configurable defaults (fixes #4224, fixes #6086) (#7131) 2021-02-04 21:10:41 +01:00
greatroar
31119ed61a
lib/ignore: Store cache timestamps as Unix ns counts (#7326) 2021-02-04 18:39:06 +01:00
Simon Frei
070bf3b776
lib/db: Report number of repaired items from checkGlobal (#7329) 2021-02-04 14:42:46 +01:00
greatroar
42917d707d
lib/scanner: Remove unused field, move WaitGroup.Add out of loop (#7323) 2021-02-03 14:25:24 +01:00
Jakob Borg
3b7a57d108
lib/protocol: Hide repeated data blocks in a given file (#7319) 2021-02-02 20:15:14 +01:00
Audrius Butkevicius
a7d9268e4d
lib/model: Make /browse endpoint return sane objects (#7306) 2021-02-01 09:27:34 +01:00
Simon Frei
052dc13487
lib/model: Correct pull progress for small files (fixes #7263) (#7316) 2021-01-31 23:40:15 +01:00
Simon Frei
249bcb3a01
lib/model: Optimize rename detection in scanner (#7315) 2021-01-31 21:02:42 +01:00
greatroar
fbe52faf49
lib/scanner: Allocate structure for final partial block (#7310)
Benchmark results on Linux/amd64, using updated benchmark for old and
new:

name        old time/op    new time/op    delta
HashFile-8    88.6ms ± 1%    88.3ms ± 1%   -0.33%  (p=0.046 n=19+19)

name        old speed      new speed      delta
HashFile-8   201MB/s ± 1%   202MB/s ± 1%   +0.33%  (p=0.044 n=19+19)

name        old alloc/op   new alloc/op   delta
HashFile-8    59.4kB ± 0%    46.1kB ± 0%  -22.47%  (p=0.000 n=14+20)

name        old allocs/op  new allocs/op  delta
HashFile-8      29.0 ± 0%      27.0 ± 0%   -6.90%  (p=0.000 n=20+20)

Co-authored-by: greatroar <@>
2021-01-28 14:23:24 +01:00
greatroar
8b86171642
lib/stun: Inline util.OnDone, comment on its purpose (#7308)
Co-authored-by: greatroar <@>
2021-01-27 19:27:00 +01:00
Simon Frei
e19d6e993d
lib/fs: Cache all real-case results (fixes #7270) (#7286) 2021-01-27 19:25:34 +01:00
greatroar
ef0473c091
lib/util, lib/svcutil: Remove unused code (#7309)
Duplicates the definition in lib/svcutil.

Co-authored-by: greatroar <@>
2021-01-27 16:33:01 +01:00
greatroar
6c3e187d1d
lib/svcutil: Simplify doneService (#7303)
OnSupervisorDone no longer allocates.

Co-authored-by: greatroar <@>
2021-01-25 16:27:17 +01:00
André Colomb
e32a516b5f
lib/model: Forget pending folders no longer announced in ClusterConfig (fixes #5187) (#7205)
* lib/db: Add ExpirePendingFolders().

Use-case is to drop any no-longer-pending folders for a specific
device when parsing its ClusterConfig message where previously offered
folders are not mentioned any more.

The timestamp in ObservedFolder is stored with only second precision,
so round to seconds here as well.  This allows calling the function
within the same second of adding or updating entries.

* lib/model: Weed out pending folders when receiving ClusterConfig.

Filter the entries by timestamp, which must be newer than or equal to
the reception time of the ClusterConfig.  For just mentioned ones,
this assumption will hold as AddOrUpdatePendingFolder() updates the
timestamp.

* lib/model, gui: Notify when one or more pending folders expired.

Introduce new event type FolderOfferCancelled and use it to trigger a
complete refreshCluster() cycle.  Listing individual entries would be
much more code and probably just as much work to answer the API
request.

* lib/model: Add comment and rename ExpirePendingFolders().

* lib/events: Rename FolderOfferCancelled to ClusterPendingChanged.

* lib/model: Reuse ClusterPendingChanged event for cleanPending()

Changing the config does not necessarily mean that the
/resut/cluster/pending endpoints need to be refreshed, but only if
something was actually removed.  Detect this and indicate it through
the ClusterPendingChanged event, which is already hooked up to requery
respective endpoints within the GUI.

No more need for a separate refreshCluster() in reaction to
ConfigSaved event or calling refreshConfig().

* lib/model: Gofmt.

* lib/db: Warn instead of info log for failed removal.

* gui: Fix pending notifications not loading on GUI start.

* lib/db: Use short device ID in log message.

* lib/db: Return list of expired folder IDs after deleting them.

* lib/model: Refactor Pending...Changed events.

* lib/model: Adjust format of removed pending folders enumeration.

Use an array of objects with device / folder ID properties, matching
the other places where it's used.

* lib/db: Drop invalid entries in RemovePendingFoldersBeforeTime().

* lib/model: Gofmt.

My local gofmt did not complain here, strangely...

* gui: Handle PendingDevicesChanged event.

Even though it currently only holds one device at a time, wrap the
contents in an array under the "added" property name.

* lib/model: Fix null values in PendingFoldersChanged removed member.

* gui: Handle PendingFoldersChanged event.

* lib/model: Simplify construction of expiredPendingList.

* lib/model: Reduce code duplication in cleanPending().

Use goto and a label for the common parts of calling the DB removal
function and building the event data part.

* lib/events, gui: Mark ...Rejected events deprecated.

Extend comments explaining the conditions when the replacement event
types are emitted.

* lib/model: Wrap removed devices in array of objects as well.

* lib/db: Use iter.Value() instead of needless db.Get(iter.Key())

* lib/db: Add comment explaining RemovePendingFoldersBeforeTime().

* lib/model: Rename fields folderID and deviceID in event data.

* lib/db: Only list actually expired IDs as removed.

Skip entries where Delete() failed as well as invalid entries that got
removed automatically.

* lib/model: Gofmt
2021-01-25 10:58:10 +00:00
greatroar
6da83ac9f5
lib/util: Remove duplicate error handling code (#7299)
This is also in lib/svcutil, and never used by clients.

Co-authored-by: greatroar <@>
2021-01-24 20:19:10 +01:00
greatroar
9c88efd55f
lib/util: Don't modify input in UniqueTrimmedStrings (#7288)
Also clarified the comment.
2021-01-16 17:39:15 +01:00
Jakob Borg
ffcb57580f
cmd/syncthing: Provide early startup for config service (ref #7188) (#7285) 2021-01-16 12:58:02 +01:00
Simon Frei
f63cdbfcfa
lib: Apply config changes sequentially (ref #5298) (#7188) 2021-01-15 15:43:34 +01:00
Simon Frei
b2d82da20d
lib/model: Pull when folder leaves error state (fixes #7280) (#7281) 2021-01-14 13:29:01 +01:00
Jakob Borg
36acaf5e21
lib/fs: Avoid blocking new caseFs creation while waiting to drop cache (fixes #7273) (#7275) 2021-01-13 17:45:29 +01:00
Jakob Borg
253049a4cc
lib/api, lib/model: Avoid contention on filesystem for DB status call (ref #7270) (#7271)
This splits the ignore getting to two methods, one that loads from disk
(the old one) and one that just returns whatever is already loaded (the
new one). The folder summary service which is just interested in stats
now uses the latter method. This means that it, and API calls that call
it, does not get blocked by folder I/O.
2021-01-12 16:25:21 +01:00
Jakob Borg
8f199e12b3
lib/fs, lib/model: Reduce lock contention on NewCaseFilesystem (fixes #7268) (#7269) 2021-01-12 16:22:21 +01:00
greatroar
f6fac3e949
lib/ur: Plug file descriptor leak in Linux memorySize (#7266) 2021-01-11 15:15:21 +01:00
Jakob Borg
0b193b76c2
lib/config, lib/connections: Add optional connection limits (fixes #7176) (#7223)
This adds two new configuration options:

    // The number of connections at which we stop trying to connect to more
    // devices, zero meaning no limit. Does not affect incoming connections.
    ConnectionLimitEnough int

    // The maximum number of connections which we will allow in total, zero
    // meaning no limit. Affects incoming connections and prevents
    // attempting outgoing connections.
    ConnectionLimitMax int

These can be used to limit the number of concurrent connections in
various ways.
2021-01-11 15:14:44 +01:00
Jakob Borg
b13b15758d
lib/connections, lib/model: Track last connection duration (ref #7223) (#7242)
This adds a statistic to track the last connection duration per device.
It isn't used for much in this PR, but it's available for #7223 to use
in deciding how to order device connection attempts (deprioritizing
devices that just dropped our connection the last time).
2021-01-05 17:45:07 +01:00
Simon Frei
c48eb4241a
lib/model: Fix child-check when deleting dirs in pull (#7236) 2021-01-02 21:40:37 +01:00
Simon Frei
0f8290485e
lib/model: Handle index sender terminating due to error (fixes #7231) (#7232) 2020-12-30 09:59:11 +01:00
André Colomb
3d1edd2492
lib/fs: Fix TestChmodDir depending on umask (fixes #6551) (#7241)
The test would fail if the umask on UNIX is greater than 0022, because
the OS transparently subtracts it from the mode passed to Mkdir(), as
the Go documentation confirms.

Our goal here is not to test os.Mkdir(), so just make sure the desired
mode is actually set by forcing it afterwards.
2020-12-30 09:56:10 +01:00
Simon Frei
a05dc6cc47
lib/model: Cleanup redundant filesystem variables in folders (#7237) 2020-12-27 22:26:25 +01:00
Simon Frei
f13e6ca631
lib/model: Remove obsolete return val from ccHandleFolders (ref #6443) (#7229) 2020-12-23 13:10:08 +01:00
Simon Frei
a20a5f61f0
lib/ur: Send unreported failures on shutdown (#7164) 2020-12-22 20:17:14 +01:00
Jakob Borg
05f25e600e
lib/connections: Refactor connection loop (#7177)
This breaks out some methods from the connection loop to make it simpler
to manage and understand.

Some slight simplifications to remove the `seen` variable (we can filter
`nextDial` based on times are in the future or not, so we don't need to
track `seen`) and adding a minimum loop interval (5s) in case some
dialer goes haywire and requests a 0s redial interval or such.

Otherwise no significant behavioral changes.
2020-12-21 16:40:13 +01:00
Simon Frei
a744dee94c
lib/fs: Correct wrapping order for meaningful log-caller (#7209) 2020-12-21 13:01:34 +01:00
Simon Frei
78bd0341a8
all: Handle errors opening db/creating file-set (ref #5907) (#7150) 2020-12-21 12:59:22 +01:00
Simon Frei
c845e245a1
lib: Close underlying conn in protocol (fixes #7165) (#7212) 2020-12-21 11:40:51 +01:00
Simon Frei
4a787986cd
lib/db: Prevent IndexID creation race (#7211) 2020-12-21 11:32:59 +01:00
Simon Frei
bd0c9913cf
lib/db: Remove index ids when dropping folder (#7200) 2020-12-21 11:10:59 +01:00
Simon Frei
d904dfa191
lib/model: Fix flaky test and add some scanning debug (#7214) 2020-12-20 18:13:35 +01:00
Simon Frei
fa40ccece1
lib: Consistently set suture logging (#7202) 2020-12-18 19:44:00 +01:00
Simon Frei
7919310dc6
lib/model: Unflake TestIgnoreDeleteUnignore (#7208) 2020-12-18 18:42:09 +01:00
Simon Frei
739e99c4d9
lib/config: Remove deprecated pending entries from config (ref #6443) (#7204) 2020-12-17 22:49:29 +01:00
André Colomb
7502997e7e
all: Store pending devices and folders in database (fixes #7178) (#6443) 2020-12-17 19:54:31 +01:00
Jakob Borg
4142a431b5 model: Actually print folder description in "Overriding" log message 2020-12-12 12:32:24 +01:00
Simon Frei
0db3b7a530
build: Switch to gopsutil's v3 module (#7191) 2020-12-10 16:43:15 +01:00
Jakob Borg
ec5a5d5218
lib/api: Returns tags in version as list (#7190) 2020-12-10 12:22:09 +01:00
Jakob Borg
e9b68a224c
lib/connections: Handle QUIC not being available (#7186)
This does two things:

- Exclude QUIC from go1.16 builds, automatically, for now, since it
  doesn't work and just panics.

- Provide some fake listeners and dialers when QUIC is disabled.

These fake listeners and dialers indicate that they are disabled and
unsupported, which silences "Dialing $address: unknown address scheme:
quic" type of stuff which is not super helpful to the user.
2020-12-09 19:23:50 +01:00
Simon Frei
8fd6b1d428
lib/protocol: Handle slashified paths in IsEncryptedParent (fixes #7184) (#7187) 2020-12-09 18:16:14 +01:00
Simon Frei
cccbb0bd5e
lib/ur: Reset timer when there's nothing to report (#7169) 2020-11-28 20:09:22 +01:00
Simon Frei
240ae0c14f
lib/model: Unflake TestRequestReceiveEncryptedLocalNoSend (#7167) 2020-11-27 20:53:03 +01:00
Simon Frei
bbb22c8c80
lib/protocol: Send Close message on read error (#7141) 2020-11-27 11:31:20 +01:00
Simon Frei
a9764fc16c
lib: Skip deleted, locally changed on recv-enc folders (fixes #7153) (#7154) 2020-11-27 11:26:36 +01:00
Audrius Butkevicius
af13f0cd35
lib/stun: Don't notify about address changes if it's not useful (fixes #7144) (#7159) 2020-11-27 07:34:30 +01:00
Simon Frei
e1b958284e
lib/api: Shut the api down gracefully (fixes #7138) (#7157) 2020-11-26 15:49:39 +01:00
Simon Frei
bf7d03d029
lib/model: Fix enc file size when pulling (fixes #7152) (#7155) 2020-11-25 22:57:25 +01:00
Simon Frei
3169212046
lib/db: Do not reset index-id when dropping device (ref #7135) (#7156)
This reverts commit 641b7aee38.
2020-11-25 22:54:05 +01:00
Simon Frei
54b50e3d52
lib/fs, lib/model: Cover more windowsyness sanitizing paths (fixes #7075) (#7158) 2020-11-25 22:52:46 +01:00
André Colomb
e6595c1ab9
lib/model: Simplify access to Folder and Device configuration. (#7151)
Use the accessor methods with a device argument instead of looking up
in a temporary map of the same data.
2020-11-24 22:20:50 +01:00
Simon Frei
100ef10d84
lib/model: Don't send locally changed on recv-enc (fixes #7137) (#7147) 2020-11-24 21:49:45 +01:00
Simon Frei
5d2c83a7e9
lib/protocol: Fix OOR panic on recv-only folders (#7143) 2020-11-23 18:37:27 +01:00
Simon Frei
8ebd893349
lib/db: Store versions for last successful db migration (#7140) 2020-11-23 18:31:32 +01:00
Simon Frei
db1f20603a
lib/model: Retain index info for new folders/devs (ref #7100) (#7133) 2020-11-20 15:53:13 +01:00
Simon Frei
24af89c8e2
all: Refactor preparing configuration (#7127) 2020-11-20 14:21:54 +01:00
Simon Frei
641b7aee38
lib/db: Reset index-id when dropping device (#7135) 2020-11-20 14:17:09 +01:00
Simon Frei
b43eccf2fe
lib/model: Never send unpaused folder without index info (#7134) 2020-11-20 14:13:50 +01:00
Simon Frei
e11b309379
lib/model, lib/util: Replace IsComplete with NoRestartErr (ref #6947) (#7126) 2020-11-18 13:43:57 +01:00
Simon Frei
2d3a535ced
lib/model: Handle cluster-config before folder start (fixes #7122) (#7123) 2020-11-17 15:30:21 +01:00
Simon Frei
9524b51708
all: Implement suture v4-api (#6947) 2020-11-17 13:19:04 +01:00
Simon Frei
e8fc465ea8
lib/config: Use correct var in MaxConcurrentIncomingRequestKiB() (#7121) 2020-11-16 16:51:51 +01:00
Jakob Borg
cca73de6a1 lib/osutil: Consider sync() best effort (fixes #7117) 2020-11-14 09:23:27 +01:00
Simon Frei
b8051fb37e
lib/model: Prevent test deadlock (#7110) 2020-11-10 12:32:07 +01:00
Simon Frei
1d3b9876f6
lib/api, lib/db: Add file debug endpoint (#7095) 2020-11-10 09:24:45 +01:00
Rahmi Pruitt
2f6a25a56f
gui: Add advance config port mapping to gui (fixes #4824) (#7017) 2020-11-10 09:24:11 +01:00
Jakob Borg
846b265430
lib/tlsutil: Add O and OU to generated certificates (fixes #7108) (#7109) 2020-11-09 17:02:56 +01:00
Simon Frei
31559e908b
all: Add untrusted folders behind feature flag (ref #62) (#7055) 2020-11-09 15:33:32 +01:00
Simon Frei
54643e86b5
lib/model: Fix locking when resending cluster-configs (#7107) 2020-11-09 14:05:21 +01:00
Simon Frei
1f1729ba43
lib/model: Add done chan to track folder-lifetime (fixes #6664) (#7094) 2020-11-09 09:05:48 +01:00
Simon Frei
d4ce0dfd84
lib/model: Send indexes for newly shared folder (fixes #7098) (#7100) 2020-11-09 08:58:46 +01:00
Simon Frei
cc9ea9db89
lib/folder: Clear pull errors when nothing is needed anymore (#7093) 2020-11-06 14:22:20 +01:00
Simon Frei
a08a1b6998
lib/api: Fix debug endpoints (ref #7001) (#7092) 2020-11-06 14:21:37 +01:00
Simon Frei
d0ccea0404
lib/config: Sanity checks on MaxConcurrentWrites (ref #7064) (#7069) 2020-11-03 19:09:32 +01:00
Simon Frei
a38b370c8d
lib/ur: Fix panics in failure-reporting (fixes #7090) (#7091) 2020-11-03 12:29:33 +01:00
Jakob Borg
7892547873
lib: Remove USE_BADGER experiment (#7089)
This removes the switch for using a Badger database, because it has bugs
that it seems there is no interest in fixing, and no actual bug tracker
to track them in.

It retains the actual implementation for the sole purpose of being able
to do the conversion back to LevelDB if anyone is actually running with
USE_BADGER. At some point in a couple of versions we can remove the
implementation as well.
2020-11-03 09:10:35 +01:00
Simon Frei
4d1bcd718c
lib/api: Fix /rest/config path and add methods to cors (ref #7001) (#7081) 2020-11-01 21:36:54 +01:00
André Colomb
7dc0c6ab43
lib/api: Allow OPTIONS method in CORS preflight request handling (ref #7017) (#7079)
This allows for checking GUI / API availability without actually doing
a GET or POST request.
2020-11-01 14:29:55 +01:00
Tomasz Wilczyński
4a616f3cb2
lib/config: Check for "msdos" when detecting FAT FS in Android (#7072) 2020-10-30 15:13:56 +01:00
Simon Frei
9189c79d74
lib/api: Add missing config mod. locks (ref #7001) (#7053) 2020-10-23 10:34:20 +02:00
Simon Frei
f0f60ba2e7
lib/api: Add /rest/config endpoint (fixes #6540) (#7001) 2020-10-22 19:54:35 +02:00
Simon Frei
1c2be84e4e
lib/model: Pass device infos as struct (fixes #7051) (#7052) 2020-10-22 13:05:31 +02:00
Simon Frei
2ba3be5e4d
lib/db: Add mechanism to repair db without schema update (ref #7044) (#7047) 2020-10-21 14:21:09 +02:00
Simon Frei
5c91723ef2
lib/model: Handle index sender lifetime (fixes #7034) (#7038) 2020-10-21 11:51:53 +02:00
Audrius Butkevicius
a17a8cd48b
lib/connections: Fix LAN addresses begin advertised even when disabled (fixes #7035) (#7045) 2020-10-21 09:16:44 +02:00
Simon Frei
27c91c57d5
lib/db: Remove need for the right dev removing globals (fixes #7036) (#7044) 2020-10-21 08:26:10 +02:00
Simon Frei
1a8c10a8d0
lib/model: Use winning version instead of merge on conflict (#6995) 2020-10-19 08:53:19 +02:00
Simon Frei
01a7ef3b0f
lib/db: Undo adding user info to panic msgs (ref #7029) (#7040) 2020-10-19 08:40:37 +02:00
Simon Frei
23c935b05a
lib/db: Ignore not found on delete in recalcGlobal (ref #7026) (#7041) 2020-10-19 08:28:53 +02:00
Simon Frei
d828adb648
cmd/stcrashreceiver, lib/db: Improve panic message handling (#7029) 2020-10-08 17:37:45 +02:00
Simon Frei
9d09fd6af3
all: Add failure reporting (fixes #7028) (#6925) 2020-10-07 10:05:13 +02:00
Simon Frei
14ae330eff
lib/db: Improve error handling checking db (ref #7026) (#7027) 2020-10-06 20:14:09 +02:00
Simon Frei
42de53c6c9
lib/model: Fix race in GlobalDirectory tests (fixes #7021) (#7022) 2020-10-03 20:46:17 +02:00
Simon Frei
48da6f0f22
lib: Use uint64 for disk stats (ref #3930) (#7019) 2020-10-02 15:22:28 +02:00
Simon Frei
a20c6ca868
lib/model, lib/protocol: Send ClusterConfig on config change (fixes #7020) (#7018) 2020-10-02 11:49:51 +02:00
Audrius Butkevicius
e027175446
all: Move remaining protos to use the vanity plugin (#7009) 2020-10-02 08:07:05 +02:00
Jakob Borg
7774932302
lib/config, lib/syncthing: Only drop delta index on upgrade if so ordered (fixes #6982) (#6983) 2020-09-30 20:16:30 +02:00
Simon Frei
1b1d38183d
lib: Remove HelloResult type, same as Hello (#7015) 2020-09-29 12:17:38 +01:00
Tomasz Wilczyński
fb3281b647
lib/fs: Add COM0 and LPT0 to invalid Windows file names (ref #7008) (#7013)
COM0 and LPT0 are not listed in the official Microsoft's documentation
at https://docs.microsoft.com/windows/win32/fileio/naming-a-file, but
in reality are also invalid in Windows Explorer.

Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
2020-09-28 20:35:54 +02:00
Jakob Borg
e6b1f67ecf
lib/fs: Be even more strict about Windows names (ref #7008) (#7012)
Things like nul.whatever.txt are also not allowed.
2020-09-28 10:42:37 +02:00
Jakob Borg
9e0b924d57
lib/fs: Be more clear about invalid file names (ref #7010) (#7011)
Add specific errors for the failures, resulting in this rather than just
the generic "invalid filename":

[MRIW7] 08:50:50 INFO: Puller (folder default, item "NUL"): syncing: filename is invalid: name is reserved
[MRIW7] 08:50:50 INFO: Puller (folder default, item "fail."): syncing: filename is invalid: name ends with space or period
[MRIW7] 08:50:50 INFO: Puller (folder default, item "sup:yo"): syncing: filename is invalid: name contains reserved character
[MRIW7] 08:50:50 INFO: default: Failed to sync 3 items
2020-09-28 10:22:50 +02:00
Jakob Borg
df99237a7f
lib/model: Disallow reserved names, and names ending with period on Windows (fixes #7008) (#7010)
Like we already do with names ending in space or containing invalid
characters.
2020-09-28 08:25:43 +01:00
Simon Frei
8452fd2ab4
lib/model, lib/scanner: Prevent races aborting scans (fixes #6994) (#6997) 2020-09-25 11:27:44 +02:00
Tobias Klauser
c390565eef
lib/fs: Use file clone ioctl wrappers and types from golang.org/x/sys/unix (#7000)
Use the IoctlFileClone and IoctlFileCloneRange ioctl wrappers and the
FileCloneRange type provided by golang.org/x/sys/unix instead of
locally implementing them. This also allows to re-enable the code for
ppc/ppc64/ppc64le again (see commit 758a1a6a37 ("lib/fs: Disable ioctl
on ppc (fixes #6898) (#6901)")) since golang.org/x/sys/unix internally
uses the correct FICLONE and FICLONERANGE values depending on $GOARCH.
2020-09-24 10:29:32 +02:00
Simon Frei
8210466b03
lib/model: Consider case conflicts when checking to be deleted items (#6986) 2020-09-12 07:45:50 +02:00
greatroar
3e24d82513
lib/fs: Optimize UnicodeLowercase (#6979)
Most notably, it now detects all-lowercase files and returns these
as-is. The tests have been expanded with two cases and are now used
as a benchmark (admittedly a rather trivial one).

name                           old time/op    new time/op    delta
UnicodeLowercaseMaybeChange-8    4.59µs ± 2%    4.57µs ± 1%    ~     (p=0.197 n=10+10)
UnicodeLowercaseNoChange-8       3.26µs ± 1%    3.09µs ± 1%  -5.27%  (p=0.000 n=9+10)
2020-09-11 09:16:10 +02:00
Simon Frei
08bebbe59b
lib/db, lib/syncthing: Don't repair DB on upgrade, but on error (fixes #6917) (#6971) 2020-09-10 10:54:41 +02:00
Simon Frei
c5c23ed10f
lib/model: Consider existing file when handling symlink on windows (#6977) 2020-09-10 10:52:38 +02:00
Simon Frei
286698ccb1
lib/model: Handle symlink deletion on windows (fixes #6972) (#6976) 2020-09-10 08:25:44 +02:00
Simon Frei
56d48d341f
lib/model: Fix case-only renames on pull (#6978) 2020-09-10 08:23:54 +02:00
Jakob Borg
780fb3bac1
lib/fs: More efficient casefs cache (#6974)
This changes the cache to cache less things, yet retain the required
efficiency for our walk usecase. This uses less memory.

Specifically, instead of keeping result and child caches for each path
level, only keep a single cached child. In practice our operations are
depth-first, or almost depth-first, and then we retain the same hit
ratio for a smaller cache size.

I improved the benchmark so that it counts the Lstat and DirNames
operations performed, and they do not change significantly. The amount
of allocated memory is reduced by 20% and the walk itself is actually
slightly faster.

This also removes the clear based on number of cached names (as that is
not a thing any more) and the timer based clear (which was unused). This
means we'll retain the last cache state forever until it's cleared by a
write operation, but we did that before too and that state is now a lot
smaller...

The overhead compared to not using a casefs, for our typical "double
walk" (walk the tree then stat everything again) is 2x the dirnames we
would otherwise call, and no overhead on the stats (unchanged from old
implementation)

```
name                         old time/op         new time/op         delta
WalkCaseFakeFS100k/rawfs-8           306ms ± 1%          305ms ± 2%     ~     (p=0.182 n=9+10)
WalkCaseFakeFS100k/casefs-8          579ms ± 5%          557ms ± 1%   -3.77%  (p=0.000 n=10+10)

name                         old B/entry         new B/entry         delta
WalkCaseFakeFS100k/rawfs-8             590 ± 0%            590 ± 0%     ~     (all equal)
WalkCaseFakeFS100k/casefs-8          1.09k ± 0%          0.87k ± 0%  -19.98%  (p=0.000 n=10+10)

name                         old DirNames/entry  new DirNames/entry  delta
WalkCaseFakeFS100k/rawfs-8            0.51 ± 0%           0.51 ± 0%     ~     (all equal)
WalkCaseFakeFS100k/casefs-8           1.02 ± 0%           1.02 ± 0%     ~     (all equal)

name                         old DirNames/op     new DirNames/op     delta
WalkCaseFakeFS100k/rawfs-8           51.2k ± 0%          51.2k ± 0%     ~     (all equal)
WalkCaseFakeFS100k/casefs-8           102k ± 0%           102k ± 0%     ~     (all equal)

name                         old Lstat/entry     new Lstat/entry     delta
WalkCaseFakeFS100k/rawfs-8            3.02 ± 0%           3.02 ± 0%     ~     (all equal)
WalkCaseFakeFS100k/casefs-8           3.02 ± 0%           3.02 ± 0%     ~     (all equal)

name                         old Lstat/op        new Lstat/op        delta
WalkCaseFakeFS100k/rawfs-8            302k ± 0%           302k ± 0%     ~     (all equal)
WalkCaseFakeFS100k/casefs-8           302k ± 0%           302k ± 0%     ~     (all equal)

name                         old allocs/entry    new allocs/entry    delta
WalkCaseFakeFS100k/rawfs-8            15.7 ± 0%           15.7 ± 0%     ~     (all equal)
WalkCaseFakeFS100k/casefs-8           27.5 ± 0%           26.1 ± 0%   -5.09%  (p=0.000 n=10+10)

name                         old ns/entry        new ns/entry        delta
WalkCaseFakeFS100k/rawfs-8           2.02k ± 1%          2.02k ± 2%     ~     (p=0.163 n=9+10)
WalkCaseFakeFS100k/casefs-8          3.83k ± 5%          3.68k ± 1%   -3.77%  (p=0.000 n=10+10)

name                         old alloc/op        new alloc/op        delta
WalkCaseFakeFS100k/rawfs-8          89.2MB ± 0%         89.2MB ± 0%     ~     (p=0.364 n=9+10)
WalkCaseFakeFS100k/casefs-8          164MB ± 0%          131MB ± 0%  -19.97%  (p=0.000 n=10+10)

name                         old allocs/op       new allocs/op       delta
WalkCaseFakeFS100k/rawfs-8           2.38M ± 0%          2.38M ± 0%     ~     (all equal)
WalkCaseFakeFS100k/casefs-8          4.16M ± 0%          3.95M ± 0%   -5.05%  (p=0.000 n=10+10)
```
2020-09-09 14:38:39 +02:00
Simon Frei
e3cd9219b8
lib/model: Don't fail over case-conflict on tempfile (fixes #6973) (#6975) 2020-09-09 11:47:14 +02:00
Simon Frei
b628ec5054
lib/fs: Use generic case resolver on windows (fixes #6968) (#6969) 2020-09-08 13:10:26 +02:00
Simon Frei
ecc24428ac
lib/model: Prevent duplicate need count on recalculating metadata (#6964) 2020-09-07 20:19:03 +02:00
Jakob Borg
674fca3868
lib/db, lib/protocol: Never need empty-version entries (fixes #6961) (#6962)
Avoid havoc when discovering locally-deleted-upgraded files during repair / need calculation...

Co-authored-by: Simon Frei <freisim93@gmail.com>
2020-09-07 20:18:25 +02:00
Simon Frei
e19728d8cc
lib/model: Correct completion perc. when globalBytes is 0 (#6963) 2020-09-07 20:03:18 +02:00
Jakob Borg
415adfbae6
lib/db: Don't hang on double iterator release with Badger (#6960)
Since iterators must be released before committing or discarding a
transaction we have the pattern of both deferring a release plus doing
it manually. But we can't release twice because we track this with a
WaitGroup that will panic when there are more Done()s than Add()s. This
just adds a boolean to let an iterator keep track.
2020-09-07 10:31:33 +02:00
Simon Frei
3dd13c3994
test, lib/model: Various integration test updates & improvements (#6956) 2020-09-07 09:35:37 +02:00
Simon Frei
52d1681fe6
lib/db: Copy returned, old FileVersion (#6952) 2020-09-04 14:13:38 +02:00
Simon Frei
7b821d2550
lib/db: Add local need check&repair (#6950) 2020-09-04 14:01:46 +02:00
Simon Frei
0e3e0a7c9e
cmd/stindex, lib/db: Use same condition to check for needed files (#6954) 2020-09-04 13:59:04 +02:00
Jakob Borg
71bfad0bc6 cmd/stindex: Add dumping folder meta 2020-09-04 11:39:21 +02:00
Jakob Borg
b7986801cd cmd/stindex: Print more hashes and versions 2020-09-04 09:10:33 +02:00
Simon Frei
6365e6108f
lib/model: Don't acquire I/O token for send-only pull (#6951) 2020-09-03 13:59:45 +02:00
Simon Frei
540518a7b7
lib/model: Add missing lock on download-state (fixes #6880) (#6945) 2020-08-30 08:03:10 +02:00
Jakob Borg
3ffbe45a6d
lib/db, lib/model: Avoid accounting mishap on folder restart (fixes #6938) (#6939)
We created a new fileset before stopping the folder during restart. When
we create that fileset it loads the current metadata and sequence
numbers from the database. But the folder may have time to update those
before stopping, leaving the new fileset with bad data.

This would cause wrong accounting (forgotten files) and potentially
sequence reuse causing files not sent to other devices.

This change reuses the fileset on restart, skipping the issue entirely.
It also moves the creation of the fileset back under the lock so there
should be no chance of concurrency issues here.
2020-08-27 18:33:27 +02:00
Jakob Borg
3393db1f69
lib/model: Save config after auto accepting folders (fixes #6922) (#6937) 2020-08-27 16:28:19 +02:00
Jakob Borg
145d87ce70 lib/model: Fixup spelling of protocol.FileIntf in test (ref #6463) 2020-08-27 16:21:04 +02:00
Simon Frei
06ac631351
lib/model: Reduce fmut locking in ClusterConfig (#6913)
The FileSet.Drop operation in there needs to potentially update a whole lot of global lists, which can take a while (longer than the deadlock interval apparently)
2020-08-27 16:01:46 +02:00
Simon Frei
1fc2dbdeeb
lib/model: Add test for recv-only changes undone by remote (#6463)
Co-authored-by: Audrius Butkevicius <audrius.butkevicius@gmail.com>
2020-08-27 15:54:00 +02:00
Simon Frei
3f0eba388c
lib/syncthing: Also cleanup on startup error (#6926) 2020-08-27 15:52:51 +02:00
Audrius Butkevicius
b19b5c95d3
lib/connections: Announce LAN addresses by default (fixes #6928) (#6896) 2020-08-25 11:48:14 +02:00
Audrius Butkevicius
d507d932b8
all: Use protobuf to generate config structs (fixes #6734) (#6900) 2020-08-25 08:11:14 +02:00
Rahmi Pruitt
5b953033c7
Created cleanup functionality for syncthing (#6884)
* Add clean up for Simple File Versioning pt.1

created test

* Add clean up for Simple File Versioning pt.2

Passing the test

* stuck on how javascript communicates with backend

* Add trash clean up for Simple File Versioning

Add trash clean up functionality of to allow the user to delete backups
after specified amount of days.

* Fixed html and js style

* Refactored cleanup test cases

Refactored cleanup test cases to one file and deleted duplicated code.

* Added copyright to test file

* Refactor folder cleanout to utility function

* change utility function to package private

* refactored utility function; fixed build errors

* Updated copyright year.

* refactor test and logging

* refactor html and js

* revert style change in html

* reverted changes in html and some js

* checkout origin head version edit...html

* checkout upstream master and correct file
2020-08-24 12:14:30 +01:00
Simon Frei
a65b46debd
lib/model: Don't mark globally deleted items as recv-only (fixes #6910) (#6911) 2020-08-21 12:34:22 +02:00
Simon Frei
b628460930
lib/model: Prevent localflag flipflopping on del. item (ref #6865) (#6919) 2020-08-21 10:09:48 +02:00
Simon Frei
cc1f6e4d4a
lib/db, lib/model: Cover exec-paths with debug logging (#6918) 2020-08-20 16:11:20 +02:00
Simon Frei
88599bc154
lib/model: Handle del. dir with locally changed items on pull (fixes #6873) (#6914) 2020-08-20 10:56:29 +02:00
Simon Frei
fc2c46e82f
lib/config, lib/fs: Make junction behaviour configurable (ref #6606) (#6907) 2020-08-19 19:58:51 +02:00
Simon Frei
ce4d149bf5
lib/nat: Don't hang on draining timer chan (fixes #6908) (#6912) 2020-08-19 15:58:44 +01:00
Audrius Butkevicius
cbbc262161
lib/db: Dont recurse on flush (fixes #6905) (#6906)
Or else we crash.

Co-authored-by: Jakob Borg <jakob@kastelo.net>
Co-authored-by: Simon Frei <freisim93@gmail.com>
2020-08-19 13:13:44 +02:00
Audrius Butkevicius
758a1a6a37
lib/fs: Disable ioctl on ppc (fixes #6898) (#6901) 2020-08-19 07:56:35 +02:00
Audrius Butkevicius
d0c6c18b4f
lib/dialer: Try dialing without reuse in parallel (fixes #6892) (#6893) 2020-08-18 13:11:12 +02:00
Audrius Butkevicius
bf9ff17267
all: Remove need to restart syncthing (#6883) 2020-08-18 09:26:33 +02:00
Simon Frei
8f5215878b
lib/db: Don't put truncated files (ref #6855, ref #6501) (#6888) 2020-08-18 09:20:12 +02:00
Simon Frei
9c2117f08e
lib/osutil: Check returned error instead of info (ref #6885) (#6887) 2020-08-12 11:33:56 +02:00
Simon Frei
fd8bea6179
lib/osutil: Preserve perms in AtomicWriter (fixes #tbd) (#6885) 2020-08-10 18:48:37 +02:00
Audrius Butkevicius
e9bb17307d
lib/fs: Unwrap mtimeFile, get fd the "correct" way (ref #6875) (#6877) 2020-08-07 07:47:48 +02:00
Jakob Borg
5e1f39b6f6 lib/fs: Fix WatchRename test for FreeBSD (fixes #6613) 2020-08-03 23:24:01 +02:00
Audrius Butkevicius
96e197e502
lib/model: Don't close file early (fixes #6875) (#6876) 2020-08-03 21:54:42 +02:00
Simon Frei
bbf25e2d02
lib/db: Log context on panic (#6872) 2020-08-01 17:32:36 +02:00
Simon Frei
a85bc1c3a6
lib/model: Check folder error before sync-waiting (fixes #6793) (#6847) 2020-07-31 19:26:09 +02:00