Commit Graph

2079 Commits

Author SHA1 Message Date
Jakob Borg
59b1b0e1dc
lib/osutil: Fix "atomic" rename on Windows (ref #6495, ref #6493) (#6510)
So, in a funny plot twist, it turns out that WriteFile in Go 1.13
doesn't actually set the read only bit on Windows when called with
permissions 0444 so my test was broken. With an improved test it turns
out that Rename does not, in fact, overwrite a read-only file on
Windows. This adds a fix for that.

(Rename might get improved in Go 1.15...)
2020-04-07 15:38:55 +02:00
Simon Frei
df318ed370
lib/db: Don't get blocklists on drop and missing continue (ref #6457) (#6502) 2020-04-07 13:13:18 +02:00
Simon Frei
07ce3572a0
lib/ignore: Only skip for toplevel includes (fixes #6487) (#6508) 2020-04-07 10:23:38 +02:00
Jakob Borg
7956e7d0ef
lib/{fs,scanner}: gofmt from Go 1.14 (#6509) 2020-04-07 09:31:29 +02:00
Jakob Borg
88cabb9e0a
lib/ur: Correct freaky error handling (fixes #6499) (#6500) 2020-04-06 09:53:37 +02:00
Jakob Borg
7505ea79a0
lib/osutil: Don't remove before rename on Windows (ref #6493) (#6495)
This was needed in ancient times but not currently.
2020-04-04 14:17:16 +02:00
Jakob Borg
362da59396 cmd/strelaypoolsrv: Expose check error to client, fix incorrect response code handling 2020-04-04 13:31:42 +02:00
Jakob Borg
48f9d323fa
lib/api: Add LDAP search filters (fixes #5376) (#6488)
This adds the functionality to run a user search with a filter for LDAP
authentication. The search is done after successful bind, as the binding
user. The typical use case is to limit authentication to users who are
member of a group or under a certain OU. For example, to only match
users in the "Syncthing" group in otherwise default Active Directory
set up for example.com:

    <searchBaseDN>CN=Users,DC=example,DC=com</searchBaseDN>
    <searchFilter>(&amp;(sAMAccountName=%s)(memberOf=CN=Syncthing,CN=Users,DC=example,DC=com))</searchFilter>

The search filter is an "and" of two criteria (with the ampersand being
XML quoted),

- "(sAMAccountName=%s)" matches the user logging in
- "(memberOf=CN=Syncthing,CN=Users,DC=example,DC=com)" matches members
  of the group in question.

Authentication will only proceed if the search filter matches precisely
one user.
2020-04-04 11:33:43 +02:00
Simon Frei
32245435e2
lib/model: Handle deleted items on RO for remote removes (fixes #6432) (#6464) 2020-04-02 16:14:25 +02:00
greatroar
2f26a95973
lib/db, lib/model: Code simplifications (#6484)
NamespacedKV.prefixedKey is still small enough to be inlined.
2020-03-31 14:32:24 +02:00
Simon Frei
123941cf62
lib/fs: Basic with empty root shouldn't point at / (#6483) 2020-03-31 13:56:07 +02:00
Jakob Borg
9c67d57c28
lib/api: Update ldap package (fixes #6479) (#6481) 2020-03-31 09:56:04 +02:00
mv1005
5b3466dc6e
lib/versioner: Extended tests of intervals (#6462) 2020-03-31 00:14:05 +02:00
Michael Rienstra
bca6854c03
lib/versioner: Fix "30 days" interval (fixes #6410) (#6461) 2020-03-30 23:28:53 +02:00
greatroar
c930b2e9e2
lib/rand: Various fixes (#6474) 2020-03-30 23:26:28 +02:00
greatroar
d7a257b391
lib/util: Fix potential data race (#6477)
Co-authored-by: greatroar <@>
2020-03-30 14:10:08 +01:00
greatroar
1e2379df1b
lib/protocol: faster Luhn algorithm and better testing (#6475)
The previous implementation was very generic; its tests didn't cover the
actual alphabet for device IDs.

Benchmark results on amd64:

name         old time/op    new time/op     delta
Luhnify-8      1.00µs ± 1%     0.28µs ± 4%   -72.38%  (p=0.000 n=9+10)
Unluhnify-8     992ns ± 2%      274ns ± 1%   -72.39%  (p=0.000 n=10+9)
2020-03-29 22:28:04 +02:00
greatroar
ea5c9176e1
lib/protocol: Remove unused channel Connection.preventSends (#6473)
Co-authored-by: greatroar <@>
2020-03-29 17:09:53 +01:00
greatroar
cc1b003f21
lib/weakhash: Fix speed reporting in benchmark (#6470) 2020-03-29 17:07:25 +02:00
greatroar
1c47fae206
lib/ur: Use sysctl syscall to get RAM size on Mac (#6468) 2020-03-29 14:28:46 +02:00
Simon Frei
79a758be3c
lib/model: Do Revert/Override synchronously (#6460) 2020-03-27 13:05:09 +01:00
Simon Frei
c7cf3ef899
lib/syncthing: Save version to db after upgrade ops are done (ref #6457) (#6458) 2020-03-26 16:58:21 +01:00
Simon Frei
b7dffc051e
lib/model: Remove unused func (#6456) 2020-03-26 14:19:26 +01:00
Simon Frei
7f23de4f03
all: Pass db intervals as args not env vars (#6448) 2020-03-24 13:53:20 +01:00
Jakob Borg
ca89f12be6
lib/api: Set ServerName on LDAPS connections (fixes #6450) (#6451)
tls.Dial needs it for certificate verification.
2020-03-24 12:56:43 +01:00
Simon Frei
ddfa82e990
lib/model: Unset local flag on deleted files (fixes #6436) (#6449) 2020-03-24 12:51:17 +01:00
Simon Frei
d6b4873eed
gui, lib/model: Fix for folder stats with r-o and ignoreDel (fixes #6430) (#6431) 2020-03-22 11:46:42 +01:00
Jakob Borg
c4abe6f815
lib/db: Don't whack blocks when putting truncated file (#6434)
As of the latest database checker we are again putting files without
blocks. I'm not 100% convinced that's a great idea, but we also do it
for ignored files apparently so it looks like we probably should support
it. This adds an escape hatch that must be manually enabled...
2020-03-20 12:07:14 +01:00
Simon Frei
74706bb02b
lib/db, lib/syncthing: Repair db once on upgrade (ref #6425, #6427) (#6429) 2020-03-19 15:58:32 +01:00
Simon Frei
cf11fa4327
lib/db: Fix removeFromGlobal and no filenames in error (fixes #6427) (#6428) 2020-03-19 14:32:22 +01:00
Simon Frei
40580d8b9b
lib/db: Remove emptied global list in checkGlobals (fixes #6425) (#6426) 2020-03-19 14:30:20 +01:00
Simon Frei
e25e71cdde
cmd/syncthing, lib/locations: Separate data and config dirs (fixes #4924) (#6309) 2020-03-18 20:58:11 +01:00
Simon Frei
00b2340f9a
lib/db: Checkpoint during schema updates (fixes #6422) (#6424) 2020-03-18 20:33:43 +01:00
Simon Frei
cc2a55892f
lib: Repair sequence inconsistencies (#6367) 2020-03-18 17:34:46 +01:00
Jakob Borg
80107d5f5e
lib/config: Correct spelling of address in LDAP config (#6420)
Literally noone uses this so I don't see a need to call this out or
trigger a 1.5 release for it.
2020-03-18 10:44:00 +00:00
Alex Xu
1e68ab3f90
lib/beacon: Only send to appropriately flagged interfaces (ref #5957) (#6404)
saves some traffic (potential mobile wakeups), may help with #5957.
2020-03-17 09:40:37 +01:00
Jakob Borg
e053db6a5e lib/protocol: Zero pad index ID strings 2020-03-17 07:40:52 +01:00
Simon Frei
1bd4ea0cbb
lib/db: Don't ignore failures unmarshaling version lists (#6411) 2020-03-16 10:09:27 +01:00
Simon Frei
a1cb1d70c4
lib/db: Use need func in withNeed and simplify (#6412) 2020-03-16 08:45:45 +01:00
Simon Frei
16698b12b1
lib/db: Extend set test with second remote (#6402) 2020-03-11 08:15:45 +01:00
Jakob Borg
20aaa5927b
lib/protocol: Use BlocksHash to compare block lists when available (#6401)
This is an optimization for faster equal checks on block lists.
2020-03-10 14:46:49 +01:00
Jakob Borg
d612c35290 lib/api: Ignore that one file that always shows up in git status 2020-03-07 11:46:54 +01:00
Jakob Borg
2faa1ad360
lib/db: Be more lenient during migration (fixes #6397) (#6398) 2020-03-06 20:50:55 +01:00
Jakob Borg
c7d6a6d780
gui, lib/api: Remove CPU & RAM measurements (fixes #6249) (#6393) 2020-03-04 20:27:48 +01:00
Jakob Borg
dd92b2b8f4
all: Tweak error creation (#6391)
- In the few places where we wrap errors, use the new Go 1.13 "%w"
  construction instead of %s or %v.

- Where we create errors with constant strings, consistently use
  errors.New and not fmt.Errorf.

- Remove capitalization from errors in the few places where we had that.
2020-03-03 22:40:00 +01:00
Jakob Borg
c08e253e7c
lib/db: Prevent GC concurrently with migration (fixes #6389) (#6390) 2020-02-29 19:51:32 +01:00
Jakob Borg
5de6f6d349
lib/db: Correct metadata recalculation (fixes #6381) (#6382)
If we decide to recalculate the metadata we shouldn't start from
whatever we loaded from the database, as that data is wrong. We should
start from a clean slate.
2020-02-28 11:16:33 +01:00
Jakob Borg
883497966e lib/db: Remove reference to env var that never existed 2020-02-27 11:21:35 +01:00
Jakob Borg
4f7a77597e
lib/db: Slightly improve indirection (ref #6372) (#6373)
I was working on indirecting version vectors, and that resulted in some
refactoring and improving the existing block indirection stuff. We may
or may not end up doing the version vector indirection, but I think
these changes are reasonable anyhow and will simplify the diff
significantly if we do go there. The main points are:

- A bunch of renaming to make the indirection and GC not about "blocks"
  but about "indirection".

- Adding a cutoff so that we don't actually indirect for small block
  lists. This gets us better performance when handling small files as it
  cuts out the indirection for quite small loss in space efficiency.

- Being paranoid and always recalculating the hash on put. This costs
  some CPU, but the consequences if a buggy or malicious implementation
  silently substituted the block list by lying about the hash would be bad.
2020-02-27 11:19:21 +01:00
Jakob Borg
4e4b9a872a
lib/dialer: Preserve nilness in error handling (fixes #6368) (#6369)
Also the call site where it shouldn't anyway be looking at the conn when
the err is non-nil.
2020-02-26 13:16:18 +01:00
Audrius Butkevicius
71aecc5cd4
lib/dialer: Bring back address faking connection (fixes #6289) (#6363) 2020-02-26 12:37:23 +01:00
Simon Frei
680b0b14db
lib/connections: Refactor status for testing (ref #6361) (#6362) 2020-02-25 21:18:31 +01:00
Jakob Borg
55238e3b5b
lib/connections: Actually record connection errors (#6361) 2020-02-25 16:56:24 +01:00
Simon Frei
f0e33d052a
lib: More contextification (#6343) 2020-02-24 21:57:15 +01:00
Jakob Borg
10cb14fcb8 lib/db: Allow put partial FileInfo without blocks (ref #6353) 2020-02-22 17:44:34 +01:00
Simon Frei
32e12abb43
lib/db: Don't panic on incorrect BlocksHash (fixes #6353) (#6355) 2020-02-22 16:51:23 +01:00
Simon Frei
6489feb1d7
lib/db: Schema update to repair sequence index (ref #6304) (#6350) 2020-02-22 09:36:59 +01:00
Simon Frei
fae7425bbf
lib: Modify FileInfos consistently (ref #6321) (#6349) 2020-02-19 16:58:09 +01:00
Tyler Kropp
4026625c2d
lib/config, gui: Set unix socket permissions for GUI listen address (fixes #5979) (#6310) 2020-02-18 08:52:12 +01:00
Jakob Borg
bb375b1aff
lib/model: Stop summary sender faster (ref #6319) (#6341)
One of the causes of "panic: database is closed" is that we try to send
summaries after it's been closed. Calculating summaries can take a long
time and if we have a lot of folders it's not unreasonable to think
that we might be stopped in this loop, so prepare to bail here.

* push
2020-02-14 08:11:54 +01:00
Simon Frei
05e23f1991
lib/db: Don't call backend.Commit twice (ref #6337) (#6342) 2020-02-14 08:11:24 +01:00
Jakob Borg
71de6fe290
lib/upnp: Exit quicker (#6339)
During NAT discovery we block for 10s (NatTimeoutS) before returning.
This is mostly noticeable when Ctrl-C:ing a Syncthing directly after
startup as we wait for those ten seconds before shutting down. This
makes it check the context a little bit more frequently.
2020-02-13 15:39:36 +01:00
Jakob Borg
6a840a040b
lib/db: Keep metadata better in sync (ref #6335) (#6337)
This adds metadata updates to the same write batch as the underlying
file change. The odds of a metadata update going missing is greatly
reduced.

Bonus change: actually commit the transaction in recalcMeta.
2020-02-13 15:23:08 +01:00
Simon Frei
c3637f2191
lib: Faster termination on exit (ref #6319) (#6329) 2020-02-13 14:43:00 +01:00
Simon Frei
ca90f4e6af
lib/db: Use flags from arg not LocalFlags() updating meta (#6333) 2020-02-13 14:02:30 +01:00
Jakob Borg
51fa36d61f
lib/db: Recover sequence number and metadata on startup (fixes #6335) (#6336)
lib/db: Recover sequence number and metadata on startup (fixes #6335)

If we crashed after writing new file entries but before updating
metadata in the database the sequence number and metadata will be wrong.
This fixes that.
2020-02-13 13:05:26 +01:00
Jakob Borg
d95a087829
lib/db: Don't leak snapshot when closing (#6331)
We could potentially get a snapshot and then fail to get a releaser,
leaking the snapshot. This takes the releaser first and makes sure to
release it on snapshot error.
2020-02-12 12:00:17 +01:00
Jakob Borg
a728743c86
lib/db: Use Commit() instead of commit() (#6330)
The readWriteTransaction offered both commit() (the one to use) and
Commit() (via embedding) where the latter didn't close the read
transaction. This removes the lower cased variant in order to prevent
the mistake.

The only place where the mistake was made was the new gc runner, where
it would leave a read snapshot open forever.
2020-02-12 11:59:55 +01:00
Simon Frei
ce27780a4c
lib/model: Return empty summary on paused folders (ref #6272) (#6326) 2020-02-12 11:59:12 +01:00
Jakob Borg
0df39ddc72
lib/fs, lib/model: Rewrite RecvOnly tests (#6318)
During some other work I discovered these tests weren't great, so I've
rewritten them to be a little better. The real changes here are:

- Don't play games with not starting the folder and such, and don't
  construct a fake folder instance -- just use the one the model has. The
  folder starts and scans but the folder contents are empty at this point
  so that's fine.

- Use a fakefs instead of a temp dir.

- To support the above, implement a fakefs option `?content=true` to
  make the fakefs actually retain written content. Use sparingly,
  obviously, but it means the fakefs can usually be used instead of an
  on disk real directory.
2020-02-12 07:47:05 +01:00
Simon Frei
a596e5e2f0
lib/model: Consistent error return values for folder methods on model (#6325) 2020-02-12 07:35:24 +01:00
Jakob Borg
04e648fee6
lib/db: Handle missing block lists as missing file (ref #6321) (#6322)
Also explicitly handle non-nil but empty block lists (if they should
ever pop up as an effect of unmarshalling changes or whatnot).
2020-02-11 15:37:22 +01:00
Simon Frei
29736b1e33
lib/db: Add closeWaitGroup to allow async operation (#6317) 2020-02-11 14:31:43 +01:00
Jakob Borg
b61da487e4 all: Update metadata modtime on delete (ref #6284) (#6315)
This records the time a file was marked as deleted. It will, in the
future, aid in garbage collecting old files.
2020-02-10 10:48:30 +01:00
Jakob Borg
5823e7a5ce lib/model: Sort and group model initialization better (ref #6303) 2020-02-01 08:12:25 +01:00
Jakob Borg
55937b61ca
lib/model: Add global request limiter (fixes #6302) (#6303)
This adds a new config with the simple and concise name
maxConcurrentIncomingRequestKiB. This limits how many bytes we have "in
the air" in the form of response data being read and processed.

After some testing I think that not having this limiter is seldom a
great idea and thus I propose a default value of 256 MiB for this new
setting.

I also refactored the folder IO limiter to be a model/folder attribute
instead of a package global.
2020-02-01 08:02:18 +01:00
Jakob Borg
e0d4cdc9a3
lib/ignore: Don't crash on empty patterns (fixes #6300) (#6301)
Instead of panicking when we try to look closer at the empty pattern,
return something like `invalid pattern "(?d)" in ignore file: missing
pattern`.
2020-01-30 09:58:44 +01:00
Jakob Borg
d91c4b010b
lib/config, lib/model: Limit concurrent pulls (fixes #5914) (#6290)
Adds a new folder state "Waiting to Sync" in the same vein as the
existing "Waiting to Scan". This vastly improves performances in the
rare cases when there are lots and lots of folders operating.
2020-01-27 17:31:17 +01:00
Jakob Borg
84920bff63 lib/db: Fixup last commit with better key name 2020-01-26 15:22:21 +01:00
Jakob Borg
bf4c8439e8
lib/db: Configurable block GC time (#6295)
Also retain the interval over restarts by storing last GC time in the
database. This to make sure that GC eventually happens even if the
interval is configured to a long time (say, a month).
2020-01-26 15:13:28 +01:00
Jakob Borg
8fc2dfad0c
lib/db: Deduplicate block lists in database (fixes #5898) (#6283)
* lib/db: Deduplicate block lists in database (fixes #5898)

This moves the block list in the database out from being just a field on
the FileInfo to being an object of its own. When putting a FileInfo we
marshal the block list separately and store it keyed by the sha256 of
the marshalled block list. When getting, if we are not doing a
"truncated" get, we do an extra read and unmarshal for the block list.

Old block lists are cleared out by a periodic GC sweep. The alternative
would be to use refcounting, but:

- There is a larger risk of getting that wrong and either dropping a
  block list in error or keeping them around forever.

- It's tricky with our current database, as we don't have dirty reads.
  This means that if we update two FileInfos with identical block lists in
  the same transaction we can't just do read/modify/write for the ref
  counters as we wouldn't see our own first update. See above about
  tracking this and risks about getting it wrong.

GC uses a bloom filter for keys to avoid heavy RAM usage. GC can't run
concurrently with FileInfo updates so there is a new lock around those
operation at the lowlevel.

The end result is a much more compact database, especially for setups
with many peers where files get duplicated many times.

This is per-key-class stats for a large database I'm currently working
with, under the current schema:

```
 0x00:  9138161 items, 870876 KB keys + 7397482 KB data, 95 B +  809 B avg, 1637651 B max
 0x01:   185656 items,  10388 KB keys + 1790909 KB data, 55 B + 9646 B avg,  924525 B max
 0x02:   916890 items,  84795 KB keys +    3667 KB data, 92 B +    4 B avg,     192 B max
 0x03:      384 items,     27 KB keys +       5 KB data, 72 B +   15 B avg,      87 B max
 0x04:     1109 items,     17 KB keys +      17 KB data, 15 B +   15 B avg,      69 B max
 0x06:      383 items,      3 KB keys +       0 KB data,  9 B +    2 B avg,      18 B max
 0x07:      510 items,      4 KB keys +      12 KB data,  9 B +   24 B avg,      41 B max
 0x08:     1349 items,     12 KB keys +      10 KB data,  9 B +    8 B avg,      17 B max
 0x09:      194 items,      0 KB keys +     123 KB data,  5 B +  634 B avg,   11484 B max
 0x0a:        3 items,      0 KB keys +       0 KB data, 14 B +    7 B avg,      30 B max
 0x0b:   181836 items,   2363 KB keys +   10694 KB data, 13 B +   58 B avg,     173 B max
 Total 10426475 items, 968490 KB keys + 9202925 KB data.
```

Note 7.4 GB of data in class 00, total size 9.2 GB. After running the
migration we get this instead:

```
 0x00:  9138161 items, 870876 KB keys + 2611392 KB data, 95 B +  285 B avg,    4788 B max
 0x01:   185656 items,  10388 KB keys + 1790909 KB data, 55 B + 9646 B avg,  924525 B max
 0x02:   916890 items,  84795 KB keys +    3667 KB data, 92 B +    4 B avg,     192 B max
 0x03:      384 items,     27 KB keys +       5 KB data, 72 B +   15 B avg,      87 B max
 0x04:     1109 items,     17 KB keys +      17 KB data, 15 B +   15 B avg,      69 B max
 0x06:      383 items,      3 KB keys +       0 KB data,  9 B +    2 B avg,      18 B max
 0x07:      510 items,      4 KB keys +      12 KB data,  9 B +   24 B avg,      41 B max
 0x09:      194 items,      0 KB keys +     123 KB data,  5 B +  634 B avg,   11484 B max
 0x0a:        3 items,      0 KB keys +       0 KB data, 14 B +   17 B avg,      51 B max
 0x0b:   181836 items,   2363 KB keys +   10694 KB data, 13 B +   58 B avg,     173 B max
 0x0d:    44282 items,   1461 KB keys +   61081 KB data, 33 B + 1379 B avg, 1637399 B max
 Total 10469408 items, 969939 KB keys + 4477905 KB data.
```

Class 00 is now down to 2.6 GB, with just 61 MB added in class 0d.

There will be some additional reads in some cases which theoretically
hurts performance, but this will be more than compensated for by smaller
writes and better compaction.

On my own home setup which just has three devices and a handful of
folders the difference is smaller in absolute numbers of course, but
still less than half the old size:

```
 0x00:  297122 items,  20894 KB keys + 306860 KB data, 70 B + 1032 B avg, 103237 B max
 0x01:  115299 items,   7738 KB keys +  17542 KB data, 67 B +  152 B avg,    419 B max
 0x02: 1430537 items, 121223 KB keys +   5722 KB data, 84 B +    4 B avg,    253 B max
 ...
 Total 1947412 items, 151268 KB keys + 337485 KB data.
```

to:

```
 0x00:  297122 items,  20894 KB keys +  37038 KB data, 70 B +  124 B avg,    520 B max
 0x01:  115299 items,   7738 KB keys +  17542 KB data, 67 B +  152 B avg,    419 B max
 0x02: 1430537 items, 121223 KB keys +   5722 KB data, 84 B +    4 B avg,    253 B max
 ...
 0x0d:   18041 items,    595 KB keys +  71964 KB data, 33 B + 3988 B avg, 101109 B max
 Total 1965447 items, 151863 KB keys + 139628 KB data.
```

* wip

* wip

* wip

* wip
2020-01-24 08:35:44 +01:00
Audrius Butkevicius
17b441c993
lib/relays: Fix incorrect timeout, bring back logging (ref #6289) (#6291)
* lib/relays: Fix incorrect timeout, bring back logging (ref #6289)

* Fix format strings
2020-01-23 21:37:35 +00:00
Simon Frei
08f0e125ef all: Transactionalize db.FileSet (fixes #5952) (#6239) 2020-01-21 18:23:08 +01:00
Jakob Borg
d62a0cf692
lib/model: Handle progress emitter zero interval (fixes #6281) (#6282)
Makes the logic a bit clearer and safer. This also sneakily redefines
the 0 interval to also mean disabled, whereas it previously meant ...
sometimes default to 1s, sometimes just spin.
2020-01-20 21:14:29 +01:00
Simon Frei
879d757850 lib/syncthing: Wait for actual termination on Stop() (#6277) 2020-01-20 08:40:15 +01:00
Jakob Borg
6c8e8f0391
lib/model: Remove legacy handling of symlinks (#6276)
This hardly seams relevant any more; 0.14.14 is dead since a long time.
2020-01-19 12:02:20 +01:00
Simon Frei
39891cdf42 lib/model: Return paused summary instead of error on paused folders (#6272) 2020-01-17 09:30:00 +01:00
Simon Frei
e782bab9fc lib/config: Add some info to the folder marker missing (ref #5207) (#6270) 2020-01-16 15:30:29 +01:00
Simon Frei
08bb730ad0 lib/db: Wrap errors from leveldb iterators (fixes #6263) (#6264) 2020-01-12 09:06:31 +04:00
Simon Frei
71882765f2 lib/events, lib/model: Unflake test and prevent deadlock on event unsubscribing (#6261) 2020-01-11 08:14:05 +01:00
Simon Frei
119d76d035
lib/stun: Refactor to remove unnecessary logging (fixes #6213) (#6260) 2020-01-10 10:24:15 +01:00
Simon Frei
08753ccabe
lib/model: Reset queue after all pulling is done (fixes #5867) (#6256) 2020-01-08 12:21:22 +01:00
Simon Frei
f56a5545d4 gui, lib/model: Prevent negative sync completion (fixes #4570) (#6248) 2020-01-03 14:07:57 +01:00
Simon Frei
7a8e73d599 build, pmp: Replace fork with upstream for go-nat-pmp and tidy go.mod (#6247) 2020-01-03 12:39:59 +01:00
Jakob Borg
a963bc8b86
lib/upgrade: Let Mac load .zip archives (#6230)
There is no need to do this switch based on the current OS, instead do
it based on what the archive actually appears to be.

(Tested; works.)
2019-12-16 07:21:18 +01:00
Simon Frei
de64ffddab lib/api: Prevent leaks in tests (#6227) 2019-12-13 09:26:41 +01:00
Simon Frei
8140350094 lib/syncthing: Expose backend instead of lowlevel (#6224) 2019-12-12 16:50:09 +01:00
Jakob Borg
325c3c1fa7
lib/db, lib/protocol: Compact FileInfo and BlockInfo alignment (#6215)
* lib/db, lib/protocol: Compact FileInfo and BlockInfo alignment

This fixes the following two lint warnings

    FileInfo: struct of size 160 bytes could be of size 136 bytes
    BlockInfo: struct of size 48 bytes could be of size 40 bytes

by reordering fields in alignment order (64 bit fields, then 32 bit
fields, then 16 bit fields (if any), then small ones). The end result is
a slightly less aesthetically pleasing struct field order, but since
these are the objects we often juggle in bulk and keep large queues of I
think it's worth it.

It's a micro optimization, but a cheap one.
2019-12-08 13:31:26 +01:00
Jakob Borg
be0508cf26
lib/model, lib/protocol: Use error handling to avoid panic on non-started folder (fixes #6174) (#6212)
This adds error returns to model methods called by the protocol layer.
Returning an error will cause the connection to be torn down as the
message couldn't be handled. Using this to signal that a folder isn't
currently available will then cause a reconnection a few moments later,
when it'll hopefully work better.

Tested manually by running with STRECHECKDBEVERY=0 on a nontrivially
sized setup. This panics reliably before this patch, but just causes a
disconnect/reconnect now.
2019-12-04 10:46:55 +01:00
Simon Frei
6fd5e78740 lib: Consistently unsubscribe from config-wrapper (fixes #6133) (#6205) 2019-12-04 07:15:00 +01:00
Paul Brit
eca156fd7f lib/osutil: Increase maxfiles on macOS properly (fixes #6206) (#6207) 2019-12-03 07:26:22 +01:00
Marcus Legendre
b3fd9a8d53 lib/ignore: Don't create empty ".stignore" files (fixes #6190) (#6197)
This will:

1. prevent creation of a new .stignore if there are no ignore patterns
2. delete an existing .stignore if all ignore patterns are removed
2019-12-02 08:19:02 +01:00
Simon Frei
0bec01b827 lib/db: Remove *instance by making everything *Lowlevel (#6204) 2019-12-02 08:18:04 +01:00
Jakob Borg
e82a7e3dfa
all: Propagate errors from NamespacedKV (#6203)
As foretold by the prophecy, "once the database refactor is merged, then
shall appear a request to propagate errors from the store known
throughout the land as the NamedspacedKV, and it shall be good".
2019-11-30 13:03:24 +01:00
Jakob Borg
928767e316 lib/model: gofmt lol :( 2019-11-29 09:29:59 +01:00
Evgeny Kuznetsov
1c277fc096 lib/fs: Add case-insensitive fakefs (#6074) 2019-11-29 09:17:42 +01:00
Jakob Borg
c71116ee94
Implement database abstraction, error checking (ref #5907) (#6107)
This PR does two things, because one lead to the other:

- Move the leveldb specific stuff into a small "backend" package that
defines a backend interface and the leveldb implementation. This allows,
potentially, in the future, switching the db implementation so another
KV store should we wish to do so.

- Add proper error handling all along the way. The db and backend
packages are now errcheck clean. However, I drew the line at modifying
the FileSet API in order to keep this manageable and not continue
refactoring all of the rest of Syncthing. As such, the FileSet methods
still panic on database errors, except for the "database is closed"
error which is instead handled by silently returning as quickly as
possible, with the assumption that we're anyway "on the way out".
2019-11-29 09:11:52 +01:00
Simon Frei
606154b183 lib/model: Also send folder summary from sync-preparing (ref #6028) (#6202) 2019-11-29 08:30:17 +01:00
Aman Gupta
a04f54a16a lib/upnp: Use simple continue in loop (#6192) 2019-11-26 22:55:34 +00:00
Aman Gupta
509d123251 lib/upnp: Ensure uPnP http requests have trailing \r\n (#6193) 2019-11-26 22:54:46 +00:00
Simon Frei
b32821a586 lib/config, lib/connections: Remove ListenAddresses hack (#6188) 2019-11-26 17:07:25 +01:00
Simon Frei
1bae4b7f50 all: Use context in lib/dialer (#6177)
* all: Use context in lib/dialer

* a bit slimmer

* https://github.com/syncthing/syncthing/pull/5753

* bot

* missed adding debug.go

* errors.Cause

* simultaneous dialing

* anti-leak
2019-11-26 07:39:51 +00:00
Jakob Borg
4e151d380c lib/versioner: Reduce surface area (#6186)
* lib/versioner: Reduce surface area

This is a refactor while I was anyway rooting around in the versioner.
Instead of exporting every possible implementation and the factory and
letting the caller do whatever, this now encapsulates all that and
exposes a New() that takes a config.VersioningConfiguration.

Given that and that we don't know (from the outside) how a versioner
works or what state it keeps, we now just construct it once per folder
and keep it around. Previously it was recreated for each restore
request.

* unparam

* wip
2019-11-26 07:39:31 +00:00
Simon Frei
f747ba6d69 lib/ignore: Keep skipping ignored dirs for rooted patterns (#6151)
* lib/ignore: Keep skipping ignored dirs for rooted patterns

* review

* clarify comment and lint

* glob.QuoteMeta

* review
2019-11-26 07:37:41 +00:00
Simon Frei
33258b06f4 lib/connections: Dialer code deduplication (#6187) 2019-11-26 07:36:58 +00:00
Simon Frei
4d368a37e2 lib/model, lib/protocol: Add contexts sending indexes and download-progress (#6176) 2019-11-25 11:07:36 +01:00
Simon Frei
cf312abc72 lib: Wrap errors with errors.Wrap instead of fmt.Errorf (#6181) 2019-11-23 15:20:54 +00:00
Simon Frei
65d4dd32cb
lib/model: Also handle ServeBackground (#6173) 2019-11-22 21:30:16 +01:00
Simon Frei
de886b3f22 lib/relay: Prevent lock nil deref when creation dynamic client (#6175) 2019-11-21 17:45:06 +00:00
Simon Frei
57d668ed1d
lib/config: Do introductions in a single config change (#6162) 2019-11-21 08:41:41 +01:00
Simon Frei
90d85fd0a2
lib: Replace done channel with contexts in and add names to util services (#6166) 2019-11-21 08:41:15 +01:00
Simon Frei
0d14ee4142
lib/model: Don't info log repeat pull errors (#6149) 2019-11-19 09:56:53 +01:00
Simon Frei
28edf2f5bb lib/model: Keep fmut locked while adding/starting/restarting folders (#6156) 2019-11-18 21:15:26 +01:00
Simon Frei
5edf4660e2
lib/model: Prevent cleanup-race in testing (ref #6152) (#6155) 2019-11-14 23:08:40 +01:00
Simon Frei
f80ce17497
lib/model: In tests prevent goroutine leaks and increase timeouts (#6152) 2019-11-13 10:21:54 +01:00
Simon Frei
ce72bee576 lib/model: Simplify pull error/retry logic (fixes #6139) (#6141) 2019-11-11 15:50:28 +01:00
Jakob Borg
d19b12d3fe lib/protocol: Buffer allocation when compressing (fixes #6146) (#6147)
We incorrectly gave a too small buffer to lz4.Compress, causing it to
allocate in some cases (when the data actually becomes larger when
compressed). This then panicked when passed to the buffer pool.

This ensures a buffer that is large enough, and adds tripwires closer to
the source in case this ever pops up again. There is a test that
exercises the issue.
2019-11-11 08:36:31 +00:00
Jakob Borg
1d99e5277a all: Cleanups enabled by Go 1.12 2019-11-10 10:16:10 +01:00
Jakob Borg
879f51b027 lib/tlsutil: Remove Go 1.12 TLS 1.3 beta opt-in
Go 1.13 enables this by default.
2019-11-10 09:32:48 +01:00
Pablo
9b01e64c66 gui, lib/api: Adds support for prefers-color-scheme (fixes #6115)
* gui, lib/api: Adds support for prefers-color-scheme on default theme (fixes #6115)

- Renames current default theme into a new "light" theme
- Modifies assets serving to allow getting assets from different themes

* lib/api: Serve assets from arbitrary theme when path starts with "theme-assets"

* lib/api: Moves constant out of function

* Loads light theme in browsers without support for prefers-color-scheme

* gui: Disables dark theme when printing

* Prevents repeated injection and adds support for older browsers

The CSS is always loaded if there is no support for `matchMedia`.
2019-11-08 21:44:37 +00:00
Audrius Butkevicius
65c172cd8d lib/api: Reset mtime after theme change (fixes #5810) (#6140) 2019-11-08 22:37:42 +01:00
Simon Frei
85e6a77f25 lib/model: Remove some testing deadlocks (#6138) 2019-11-08 18:53:51 +01:00
Jakob Borg
88244b0c1f lib/model: Add test for previous commit 2019-11-08 17:03:25 +01:00
Simon Frei
cd290d2d05 lib/model: Add initial deviceStatRefs on model creation (fixes #6136) (#6137)
This is a regression introduced in PR #6005 / commit
f7b2e79fdc
2019-11-08 11:32:51 +00:00
Simon Frei
bee7cce081
lib/model: Add folders on start in model (#6135) 2019-11-08 10:56:16 +01:00
Jakob Borg
6be6de4b4a lib/api: Slightly unflake TestCSRFRequired by allowing longer timeout 2019-11-07 08:14:49 +01:00
Jakob Borg
6755a9ca63 Fix bufferpool puts (ref #4976) (#6125)
* Fix bufferpool puts (ref #4976)

There was a logic error in Put() which made us put all large blocks into
segment zero, where we subsequently did not look for them.

I also added a lowest threshold, as we otherwise allocate a 128KiB
buffer when we need 24 bytes for a header and such.

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* smaller stress

* cap/len

* wip

* wip
2019-11-06 10:53:10 +00:00
Audrius Butkevicius
98a1adebe1 all: Remove dead code, fix lost msgLen checks (#6129) 2019-11-06 07:09:58 +01:00
Aman Gupta
31569debeb lib/upnp: Fix outdated comment (#6110) 2019-11-05 18:56:51 +00:00
Ruslan Yevdokymov
3b5dff3f34 lib/model: Fix removal of a marker when there are still folders referencing it (#6114) 2019-10-30 15:11:07 +00:00
Simon Frei
bbdda059bd lib/model: Check for symlinks before deleting during pull (fixes #6090) (#6100) 2019-10-22 21:55:51 +02:00
Simon Frei
72194d137c lib/scanner: Don't scan if input path is below symlink (fixes #6090) (#6101) 2019-10-22 11:12:21 +02:00
Audrius Butkevicius
c4f161d8c5 lib/connections: Rate limit quic accept loop (fixes #6081) (#6082) 2019-10-18 09:55:37 +02:00
Jakob Borg
ad2d3702ae all: Upgrade github.com/gogo/protobuf and regenerate (fixes #6085) 2019-10-18 09:53:59 +02:00
Jakob Borg
95acb26249 lib/syncthing: Fixup test after merge 2019-10-17 09:14:27 +02:00
Jakob Borg
4736cccda1
all: Update certificate lifetimes (fixes #6036) (#6078)
This adds a certificate lifetime parameter to our certificate generation
and hard codes it to twenty years in some uninteresting places. In the
main binary there are a couple of constants but it results in twenty
years for the device certificate and 820 days for the HTTPS one. 820 is
less than the 825 maximum Apple allows nowadays.

This also means we must be prepared for certificates to expire, so I add
some handling for that and generate a new certificate when needed. For
self signed certificates we regenerate a month ahead of time. For other
certificates we leave well enough alone.
2019-10-16 20:31:46 +02:00
Simon Frei
1a06ab68eb
lib/sync: Cleanly fail instead of panic in tests (#6088) 2019-10-16 10:11:11 +02:00
Simon Frei
b8907b49f9
lib/syncthing: Prevent hangup on error during startup (fixes #6043) (#6047) 2019-10-16 10:10:42 +02:00
Simon Frei
7b33294955 gui, lib/model: Add new state FolderPreparingSync (fixes #6027) (#6028) 2019-10-16 09:08:54 +02:00
Simon Frei
031684116b lib/util: Add caller info to service (ref #5932) (#5973) 2019-10-16 09:06:16 +02:00
Simon Frei
a0c9db1d09 lib/api: Unify JSON marshalling of file infos (#6087) 2019-10-15 11:25:12 +02:00
Jakob Borg
6b570ee8dc lib/upgrade: Add html_url release field 2019-10-08 09:12:00 +02:00
Jakob Borg
67b8ef1f3e
cmd/*, lib/build: Set correct LongVersion (fixes #5993) (#5997)
The relay and discosrv didn't use the new lib/build package, now they
do. Conversely the lib/build package wasn't aware there might be other
users and hard coded the program name - now it's set by the build
script
2019-10-07 13:30:25 +02:00
Lukas Lihotzki
96bb1c8e29 all, lib/logger: Refactor SetDebug calls (#6054) 2019-10-04 13:03:34 +02:00
Audrius Butkevicius
8fb576ed54
lib/model: Adjust blocks reported in usage reporting (fixes #5995) (#6037)
* lib/model: Adjust blocks reported in usage reporting (fixes #5995)

* Use variables, fix go.mod
2019-10-04 12:03:13 +01:00
Jakob Borg
5e31e6356f lib/api: Report actual listener address (fixes #6049) (#6060) 2019-10-04 11:25:41 +01:00
Jakob Borg
755e689627 lib/db: Always use small db settings on 32 bit archs (#6053) 2019-10-03 13:40:14 +01:00
Simon Frei
a2a14c8424 lib/model: Set empty version when unignoring deleted files (fixes 6038) (#6039) 2019-10-01 15:34:59 +02:00
Jakob Borg
52d80d8144 lib/fs: Improve root check (#6033)
The root check would allow things like c:\foobar\baz if the root was
c:\foo, because string wise that's a prefix. Now it doesn't.
2019-09-29 23:38:11 +08:00
Simon Frei
35b699dc77 lib/fs: Check events against both the user and eval root (#6013) 2019-09-22 08:03:22 +01:00
Jakob Borg
db0ba2555a gui, lib/api: Use effective listen address for no auth warning
This adds a field `guiAddressUsed` to the system status response, that
holds the current listening address actually in use. This may be
different from the one stored in the config because it may have been
overridden by environment or command line flag.

The GUI now checks this field to see if we are listening on localhost.
If we are not, the authentication required warning is displayed,
regardless of the *configured* listening address.
2019-09-20 16:23:33 +02:00
ghjklw
80a83b605c lib/config: Remove stun.voxgratia.org (fixes #6010) (#6011)
DNS resolution fails for this server:
named[9495]: REFUSED unexpected RCODE resolving 'stun.voxgratia.org/A/IN': 2600:9000:5303:ae00::1#53
named[9495]: REFUSED unexpected RCODE resolving 'stun.voxgratia.org/A/IN': 205.251.198.31#53
2019-09-13 09:05:25 +01:00
Simon Frei
28b6e8b063
lib/db: Update db when only local flags change (fixes #6008) (#6007) 2019-09-12 08:47:39 +02:00
Simon Frei
f7b2e79fdc lib/model: Use read-locks wherever possible (#6005) 2019-09-12 05:55:23 +01:00
Jakob Borg
22dff7207c
lib/api: Refactor to run tests in parallel (#5998)
This is an experiment in testing, based on the advise to always call
t.Parallel() at the start of every test. Doing so makes tests run in
parallel, which is usually faster, but also exposes package level state
and potential race conditions better.

To support this I had to redesign the CSRF manager to not be package
global, which was indeed an improvement. And tests run five times faster
now.
2019-09-05 12:35:51 +01:00
Jakob Borg
0104e78589
lib/connections: Improve write rate limiting (fixes #5138) (#5996)
This splits large writes into smaller ones when using a rate limit,
making them into a legitimate trickle rather than large bursts with a
long time in between.
2019-09-04 11:12:17 +01:00
Jakob Borg
80894948f6
build: Upgrade github.com/gogo/protobuf (#5994)
This is the result of:

- Changing build.go to take the protobuf version from the modules
  instead of hardcoded
- `go get github.com/gogo/protobuf@v1.3.0` to upgrade
- `go run build.go proto` to regenerate our code
2019-09-04 07:33:29 +01:00
Jakob Borg
e945e65b13
lib/api: Skip an IPv6 specific test inside Docker (fixes #5991) (#5992)
Isn't this just the most beautiful thing you've ever seen?
2019-09-03 21:14:36 +01:00
Jakob Borg
ebd2e5bf30
lib/model: Correctly handle manual rescan with ignore error (fixes #5985) (#5987)
Assume a folder error was set due to bad ignores on the latest scan.
Previously, doing a manual rescan would result in:

1. Clearing the folder error, which schedules (immediately) an fs
   watcher restart

2. Attempting to load the ignores, which fails, so we set a folder
   error and bail.

3. Now the fs watcher restarts, as scheduled, so we trigger a scan.
   Goto 1.

This change fixes this by not clearing the error until the error is
actually cleared, that is, if both the health check and ignore loading
succeeds.
2019-08-30 13:27:26 +01:00
Jakob Borg
60c07b259e lib/ignore: Don't crash in partial #include line (ref #5985) (#5986)
If the line is just "#include" with nothing following it we would crash
with an index out of bounds error. Now it's a little more careful.
2019-08-30 11:36:31 +02:00
Jakob Borg
5851aabe02 lib/upgrade: Include browser_download_url field 2019-08-29 16:10:13 +02:00
Jakob Borg
0832285d79 lib/ur: Prevent trivial race in CPU bench 2019-08-25 23:43:28 +02:00
Jakob Borg
c2ea9d119d
lib/connections: Upgrade QUIC package, use contexts for timeout (#5972) 2019-08-23 10:15:52 +02:00
Simon Frei
534f07d9ca
lib/discover: Don't leak goroutines (#5950) 2019-08-22 12:30:57 +02:00
Jakob Borg
24d4290d03 lib/model, lib/scanner: Pass a valid event logger (fixes #5970) (#5971) 2019-08-21 08:05:43 +02:00
Simon Frei
2d124e053c
test: Get integration tests up to speed (config, build and test fixes) (#5962) 2019-08-20 10:17:11 +02:00
Jakob Borg
90b70c7a16 lib/db: Use different defaults for larger databases (fixes #5966) (#5967)
This introduces a better set of defaults for large databases. I've
experimentally determined that it results in much better throughput in a
couple of scenarios with large databases, but I can't give any
guarantees the values are always optimal. They're probably no worse than
the defaults though.
2019-08-20 09:41:41 +02:00
Simon Frei
77a5980747 lib/model: Do free disk space check later on pull (fixes #5948) (#5949) 2019-08-16 09:40:53 +02:00
Simon Frei
b677464dfa lib/model: Optimise locking around conn-close and puller states (#5954) 2019-08-16 09:35:19 +02:00
Simon Frei
b1c74860e8
all: Remove global events.Default (ref #4085) (#5886) 2019-08-15 16:29:37 +02:00
Simon Frei
f6f696c6c5 lib/config: Prevent nil deref in debug logging (fixes #5955) (#5956) 2019-08-15 15:51:09 +02:00
Simon Frei
cf40ed6cec lib/connections: Return exported intf from exported function (#5947) 2019-08-13 09:33:33 +02:00
Simon Frei
6fa02d5081 lib/model: Fix a few more problematic locks (ref #5929) (#5944) 2019-08-13 09:04:43 +02:00
Simon Frei
86e35f1879 lib/model: Less locking in ClusterConfig (#5943) 2019-08-11 19:30:24 +02:00
Jakob Borg
720a6bf62e lib/tlsutil: Remove hardcoded curve preferences (fixes #5940) (#5942)
They are arguable outdated and we are better off trusting the standard
library than trying to keep up with it ourselves.
2019-08-11 19:01:57 +02:00
Simon Frei
4a619e74f2 lib/model: Fix incorrect locking (#5939) 2019-08-11 16:10:30 +02:00
Audrius Butkevicius
58ef5368f8
lib/connections: Validate device id before assuming success (fixes #5934) (#5935)
* lib/connections: Validate device id before assuming success (fixes #5934)

* Vet
2019-08-09 12:31:42 +01:00
Oliver Freyermuth
edf2399ce6 Add NATSymmetricUDPFirewall to punchable NATs (#5931)
NATSymmetricUDPFirewall actually is not NAT at all, but means the machine has a global IP address and an UDP firewall in front (RFC calls it Symmetric UDP Firewall). This is punchable fine, both theoretically and also practically in testing.
2019-08-06 12:26:02 +01:00
Jakob Borg
61b9f7bd55
lib/config: Format bytes in insufficient space errors (fixes #5920) (#5921) 2019-08-02 14:43:05 +02:00
Simon Frei
77cc87dfca lib/connections: Log errors in relay clients (#5917) 2019-08-01 17:37:58 +02:00
Simon Frei
8a06cf0973 lib/model: Unflake TestPullInvalidIgnored (#5918) 2019-08-01 11:07:41 +02:00
Simon Frei
05835ed81f all: Remove potentially problematic errors from panics (fixes #5839) (#5912) 2019-07-31 10:53:35 +02:00
Simon Frei
df522576ac lib/model: Don't call t.Fatal in goroutines (fixes #5901) (#5903) 2019-07-30 17:50:51 +02:00
Simon Frei
d681ac11fe
lib/config: Handle empty Fstype for mtime-window (#5906) 2019-07-30 15:23:00 +02:00
Simon Frei
7d5f7d508d lib/syncthing: Stop only once (fixes #5908) (#5909) 2019-07-29 20:07:19 +02:00
Simon Frei
1d182e4631 lib/fs: Use gopsutils for disk usage (#5905) 2019-07-29 20:06:17 +02:00
Simon Frei
710f5c199f lib/db: Don't use global fileset in benchmarks (#5902) 2019-07-28 22:31:24 +02:00
Simon Frei
fd847d4efe lib/model: Fix flakyness of TestRequestRemoteRenameChanged (#5904) 2019-07-28 22:29:31 +02:00
Jakob Borg
c1c976aa2b
lib/model: Don't panic on failed chmod-back on directory (fixes #5836) (#5896)
* lib/model: Don't panic on failed chmod-back on directory (fixes #5836)

This makes the "in writable dir"-wrapper log chmod-back errors instead
of panicking. To do that we need a logger so the function moved into the
model package which is also the only place it's used. The tests came
along.

(The test also exercised osutil.RenameOrCopy like some sort of
piggybacking. I removed that.)
2019-07-28 10:25:05 +02:00
Jakob Borg
159d1a68e1 lib/api: Don't log random stuff in the HTTP server (fixes #5738) (#5897) 2019-07-28 09:49:07 +02:00
Simon Frei
dd850f66bb lib/api: Close server before exiting serve (fixes #5866) (#5895) 2019-07-28 08:03:55 +02:00
Jakob Borg
669bcb748f lib/config, lib/model: Don't save on every pending folder/device update (fixes #5888) (#5890)
Wrapper methods generally don't save by themselves.
2019-07-27 11:05:00 +01:00
Jakob Borg
4e22a96602 cmd/syncthing: Print version information early (fixes #5891) (#5893) 2019-07-27 10:58:39 +01:00
Jakob Borg
a992559abc
lib/db: Add hacky way to adjust database parameters (#5889)
This adds a set of magical environment variables that can be used to
tweak the database parameters. It's totally undocumented and not
intended to be a long term or supported thing.

It's ugly, but there is a backstory. I have a couple of large
installations where the database options are inefficient or otherwise
suboptimal (24/7 compaction going on and stuff like that). I don't
*know* the correct database parameters, nor yet the formula or method to
derive them by, so this requires experimentation. Experimentation needs
to happen partly in production, and rolling out new builds for every
tweak isn't practical. This provides override points for all reasonable
values, while not changing anything by default.

Ideally, at the end of such experimentation, we'll know which values are
relevant to change and in what manner, and can provide a more user
friendly knob to do so - or do it automatically based on the database
size.
2019-07-26 22:18:42 +02:00
Simon Frei
46e72d76b5
cmd/syncthing, lib/syncthing: Create library utils (ref #4085) (#5871) 2019-07-23 23:39:20 +02:00
Simon Frei
7a4c88d4e4 lib: Add mtime window when comparing files (#5852) 2019-07-23 21:48:53 +02:00
Simon Frei
35f40e9a58 lib/model: Create new file-set after stopping folder (fixes #5882) (#5883) 2019-07-23 20:39:25 +02:00
Simon Frei
5de9b677c2
lib/fs: Fix kqueue event list (fixes #5308) (#5885) 2019-07-23 14:11:15 +02:00
Simon Frei
6f08162376
lib/model: Remove incorrect/useless panics (#5881) 2019-07-23 10:51:16 +02:00
Simon Frei
7b3d9a8dca
lib/syncthing: Refactor to use util.AsService (#5858) 2019-07-23 10:50:37 +02:00
Simon Frei
942659fb06
lib/model, lib/nat: More service termination speedup (#5884) 2019-07-23 10:49:22 +02:00
Simon Frei
b5b54ff057 lib/model: No watch-error on missing folder (fixes #5833) (#5876) 2019-07-19 19:41:16 +02:00
Simon Frei
4d3432af3e lib: Ensure timely service termination (fixes #5860) (#5863) 2019-07-19 19:40:40 +02:00
Simon Frei
1cb55904bc lib/model: Prevent panic in NeedFolderFiles (fixes #5872) (#5875) 2019-07-19 19:39:52 +02:00
Simon Frei
2b622d0774 lib/model: Close conn on dev pause (fixes #5873) (#5874) 2019-07-19 19:37:29 +02:00
Simon Frei
1894123d3c lib/syncthing: Modify exit status before stopping (fixes #5869) (#5870) 2019-07-18 20:49:00 +02:00
Jakob Borg
e7e177a6fa lib/relay: Prevent spurious relay error message (fixes #5861) (#5864) 2019-07-17 10:55:28 +02:00
Simon Frei
eed1edcca0 cmd/syncthing: Ensure myID is set by making it local (fixes #5859) (#5862) 2019-07-17 07:19:14 +02:00
Simon Frei
0025e9ccfb all: Refactor cmd/syncthing creating lib/syncthing (ref #4085) (#5805)
* add skeleton for lib/syncthing

* copy syncthingMain to lib/syncthing (verbatim)

* Remove code to deduplicate copies of syncthingMain

* fix simple build errors

* move stuff from main to syncthing with minimal mod

* merge runtime options

* actually use syncthing.App

* pass io.writer to lib/syncthing for auditing

* get rid of env stuff in lib/syncthing

* add .Error() and comments

* review: Remove fs interactions from lib

* and go 1.13 happened

* utility functions
2019-07-14 11:43:13 +01:00
Simon Frei
82b70b9fae lib/model, lib/protocol: Track closing connections (fixes #5828) (#5829) 2019-07-14 11:03:55 +02:00
Simon Frei
def4b8cee5 lib/config: Error on empty folder path (fixes #5853) (#5854) 2019-07-14 11:03:14 +02:00
Aurélien Rainone
f1a7dd766e all: Add comment to ensure correct atomics alignment (fixes #5813)
Per the sync/atomic bug note:

> On ARM, x86-32, and 32-bit MIPS, it is the caller's
> responsibility to arrange for 64-bit alignment of 64-bit words
> accessed atomically. The first word in a variable or in an
> allocated struct, array, or slice can be relied upon to be
> 64-bit aligned.

All atomic accesses of 64-bit variables in syncthing code base are
currently ok (i.e they are all 64-bit aligned).

Generally, the bug is triggered because of incorrect alignement
of struct fields. Free variables (declared in a function) are
guaranteed to be 64-bit aligned by the Go compiler.

To ensure the code remains correct upon further addition/removal
of fields, which would change the currently correct alignment, I
added the following comment where required:

     // atomic, must remain 64-bit aligned

See https://golang.org/pkg/sync/atomic/#pkg-note-BUG.
2019-07-13 14:05:39 +01:00
Simon Frei
20c8dbd9ed lib/model: Fix integer conversion (fixes #5837) (#5851) 2019-07-12 16:37:12 +02:00
xduugu
4b3f9b1af9 lib/versioner: Replace multiple placeholders in a single token in external command (fixes #5849)
* lib/versioner: Add placeholder to provide the absolute file path to external commands

This commit adds support for a new placeholder, %FILE_PATH_FULL%, to the
command of the external versioner. The placeholder will be replaced by
the absolute path of the file that should be deleted.

* Revert "lib/versioner: Add placeholder to provide the absolute file path to external commands"

This reverts commit fb48962b947358e90a76e87a7794adb6057de476.

* lib/versioner: Replace all placeholders in external command (fixes #5849)

Before this commit, only these placeholders were replaced that span a
whole word, for example "%FOLDER_PATH%". Words that consisted of more
than one placeholder or additional characters, for example
"%FOLDER_PATH%/%FILE_PATH%", were left untouched.

* fixup! lib/versioner: Replace all placeholders in external command (fixes #5849)
2019-07-12 08:45:39 +01:00
Simon Frei
3446d50201 lib/model: Remove pointless error that watch hasn't started (fixes #5833) (#5834) 2019-07-10 11:00:06 +02:00
Simon Frei
9fef1552fc lib/db, lib/model: Remove folder info from panics (ref #5839) (#5840) 2019-07-10 10:57:49 +02:00
Simon Frei
485acda63b lib/relay: Call the proper Error method (ref #5806) (#5841) 2019-07-09 22:29:19 +01:00
Simon Frei
ba056578ec
lib: Add util.Service as suture.Service template (fixes #5801) (#5806) 2019-07-09 11:40:30 +02:00
Simon Frei
863fe23347 gui, lib/model: Fix download progress accounting (fixes #5811) (#5815) 2019-06-30 09:23:47 +02:00
Simon Frei
1cf352a722 lib/model: NewFileSet outside fmut (#5818) 2019-06-29 08:49:30 +02:00
Simon Frei
b58f6ca886 lib/model: Correct/unify check if item changed (#5819) 2019-06-29 07:45:41 +02:00
Simon Frei
3c7e7e971d lib/model: Make jobQueue.Jobs paginated (fixes #5754) (#5804)
* lib/model: Make jobQueue.Jobs paginated (fixes #5754)

* fix, no test yet

* add test
2019-06-27 19:25:38 +01:00
Audrius Butkevicius
afde0727fe lib/versioner: Revert naming change (fixes #5807) (#5808) 2019-06-25 08:56:11 +03:00
Simon Frei
bf744ded31 cmd/syncthing, lib/db: Exit/close db faster (fixes #5781) (#5782)
This adds a 10s timeout on closing the db and additionally cancels active
db iterators and waits for them to terminate before closing the db.
2019-06-17 15:27:25 +03:00
Simon Frei
cea5962417 lib/model: Unflake TestPullInvalidIgnoredSR/SO (fixes #5796) (#5799) 2019-06-17 15:23:28 +03:00
Simon Frei
02752af862 lib/protocol: Don't block on Close (fixes #5794) (#5795) 2019-06-14 19:04:41 +02:00
Simon Frei
abd363e8bb
lib/model: Don't error on pulling deletion of invalid file (fixes #5791) (#5792) 2019-06-14 08:48:14 +02:00
Simon Frei
13a746e0fb lib/model: Prevent nil deref if folder stopped (fixes #5780) (#5778) 2019-06-11 11:48:51 +02:00
Audrius Butkevicius
21f50e2f8f lib/versioner: Use mtime for version cleanup (fixes #5765) (#5769) 2019-06-11 09:16:55 +02:00
Audrius Butkevicius
b7c70a9817 lib/fs: Enhance mtimefs, use everywhere (fixes #5777) (#5776) 2019-06-11 08:27:12 +02:00
Jakob Borg
42ce6be9b9
lib/ur: Implement crash (panic) reporting (fixes #959) (#5702)
* lib/ur: Implement crash (panic) reporting (fixes #959)

This implements a simple crash reporting method. It piggybacks on the
panic log files created by the monitor process, picking these up and
uploading them from the usage reporting routine.

A new config value points to the crash receiver base URL, which defaults
to "https://crash.syncthing.net/newcrash" (following the pattern of
"https://data.syncthing.net/newdata" for usage reports, but allowing us
to separate the service as required).
2019-06-11 08:19:11 +02:00
Mingxuan Lin
eb4fe808c5 lib/fs: Fallback EvalSymlinks method on windows (fixes #5609) (#5611) 2019-06-10 14:33:53 +02:00
Simon Frei
1054ce9354
lib/model: Refactor sending indexes as suture service (#5757) 2019-06-10 13:27:22 +02:00
Audrius Butkevicius
ee746263fb
lib/connections: Do not leak FDs, fix address copy (fixes #5767) (#5768)
* lib/connections: Do not leak FDs, fix address copy (fixes #5767)

* build

* Update quic_listen.go

* Update quic_listen.go
2019-06-09 22:14:00 +01:00
Jakob Borg
41ff4b323e lib/api: Correct logic for errors.jons in support bundle (fixes #5766)
The err check should always be done with != and have the error case
first, to follow the pattern and avoid mistakes like these.
2019-06-09 09:35:05 +02:00
Jakob Borg
997bb5e7e1 all: Remove "large blocks" config (#5763)
We now always use large / variable blocks.
2019-06-06 15:57:38 +01:00
Jakob Borg
ca2fa8de4e lib/build: Version 1.2 will be the Fermium Flea 2019-06-06 14:45:07 +02:00
Simon Frei
e39d3f95dd lib/protocol: Prioritize close msg and add close timeout (#5746) 2019-06-05 14:01:59 +08:00
Simon Frei
6664e01acf lib/protocol: Return from ClusterConfig when closed (#5752) 2019-05-29 12:14:00 +02:00
Jakob Borg
5ce5b2c94a
lib/config: Refactor migrations a bit (#5750)
This breaks out config migrations to a separate concept, making it
(imho) slightly easier to maintain and get an overview.
2019-05-29 11:37:44 +02:00
Audrius Butkevicius
e714df013f lib/connections: Add QUIC protocol support (fixes #5377) (#5737) 2019-05-29 09:56:40 +02:00
Simon Frei
3775a64d5c lib/protocol: Don't send anything else before cluster config (#5741) 2019-05-27 11:15:34 +01:00
Simon Frei
129df0613b lib/model: Unflake folder restart with blocking conn test (ref #5707) (#5744) 2019-05-27 10:58:09 +01:00
Simon Frei
486230768e lib/fs, lib/model: Add error channel to Watch to avoid panics (fixes #5697) (#5734)
* lib/fs, lib/model: Add error channel to Watch to avoid panics (fixes #5697)

* forgot unsupported watch

* and more non(-standard)-unixy fixes

* and windows test

* review
2019-05-25 20:08:26 +01:00
Simon Frei
9e6db72535 lib/protocol: Don't call receiver after calling Closed (fixes #4170) (#5742)
* lib/protocol: Don't call receiver after calling Closed (fixes #4170)

* review
2019-05-25 20:08:07 +01:00
Simon Frei
d91da8feee lib/model: Readd special handling of conn close in TestIssue5063 (#5743)
This partially reverts commit 64518b0f7e.
2019-05-25 18:51:13 +01:00
Simon Frei
64518b0f7e
lib/model: Close connections when model is stopped (#5733) 2019-05-25 16:00:32 +02:00
Simon Frei
5d35b2c540 lib/protocol: Test for Close on blocking send deadlock (ref #5442) (#5732) 2019-05-23 21:42:02 +01:00
Simon Frei
5b306510a0
lib/model: Consistently cleanup model in tests (#5724) 2019-05-19 14:29:07 +02:00
Simon Frei
1b2b970f32 lib/model, lib/testutils: Test closing a connection on folder restart (#5707) 2019-05-18 08:53:59 +02:00
Simon Frei
5ffbb7668d lib/model: Fix test flakyness regression (ref #5592) (#5718) 2019-05-18 08:52:50 +02:00
Simon Frei
441ea109a1
lib/model: Refactor file deletions when pulling (#5699) 2019-05-17 18:29:54 +02:00
Jakob Borg
2c866277a2 lib/api, lib/connections, gui: Show connection error for disconnected devices (fixes #3345) (#5727)
* lib/api, lib/connections, gui: Show connection error for disconnected devices (fixes #3345)

This adds functionality in the connetions service to track the last
error per address. That is in turn exposed in the /rest/system/status
API method, as that is also where we already show the listener status
from the connection service.

The GUI uses this info where it lists addresses, showing errors (if any)
in red underneath each address.

I also slightly refactored the existing status method on the connection
service to have a better name and return typed information.

* ok

* review

* formatting

* review
2019-05-16 22:11:45 +01:00
André Colomb
e4956358fb lib/model: Remove superfluous check for IndexID in remote ClusterConfig (#5717)
The check in ClusterConfig() when iterating through announced devices
in a folder explicitly skips entries without a non-zero IndexID.
Therefore, the check for IndexID == 0 just below will never be true
and the intended cleanup of local index data will not happen.

Plainly remove that check to make the intended case distinction work.
2019-05-12 21:17:55 +02:00
Simon Frei
dfbbb286fc lib/fs: Consider win83 for root path as well when watching (ref #5706) (#5709) 2019-05-11 10:06:04 +02:00
Simon Frei
fbd445fe0a lib/api: Clean up after test, enables test caching (#5710) 2019-05-11 08:41:32 +02:00
Simon Frei
2b246eeb52
lib/model: Move test utilities to separate files (#5694) 2019-05-10 13:33:45 +02:00
Simon Frei
2558b021e5 lib/fs: Remove \\?\ for drive letters when watching (fixes #5578) (#5701) 2019-05-10 09:09:58 +02:00
Jakob Borg
31be810eb6 lib/model: Don't fail operation when fsync() fails (fixes #5704) (#5705) 2019-05-09 21:20:29 +01:00
Jakob Borg
62a6d619e7 Merge branch 'release'
* release:
  lib/fs: Revert removal of \\?\ for drive letters (fixes #5695)
2019-05-08 17:38:08 +02:00
Jakob Borg
a04fcfe749 lib/fs: Revert removal of \\?\ for drive letters (fixes #5695)
This reverts commit ca823bd591 from #5633.

This un-fixes bug #5578.
2019-05-08 17:31:52 +02:00
Simon Frei
283f39ae5f lib/protocol: Revert unreleased changes related to closing connections (#5688)
This reverts commits:
    ec7c88ca55
    19b51c9b92
    5da41f75fa
    04b927104f
2019-05-08 08:08:26 +02:00
Simon Frei
b45d77b6be lib/model: Fix regression deleting directories on pull (ref #5690) (#5691) 2019-05-06 20:55:26 +02:00
Simon Frei
79e67b7f79 lib/model: Remove individual pull errors on retry (fixes #5659) (#5690) 2019-05-06 17:21:56 +02:00
Simon Frei
5fa8467756 lib/model: Consistent use of locks (#5684) 2019-05-02 18:55:39 +01:00
Simon Frei
5954b105cd
lib/model: Let fakeConnection call Model.Closed on close (#5682) 2019-05-02 14:24:55 +02:00
Simon Frei
defc5dca65
lib/model: Use correct lock (#5683) 2019-05-02 14:09:42 +02:00
Simon Frei
9f358ecae0
lib/protocol: Refactor to pass only relevant argument to writeMessage (#5681) 2019-05-02 14:09:19 +02:00
Simon Frei
fe4daf242b
cmd, lib/db: Actually close goleveldb (fixes #5505) (#5671) 2019-05-02 11:15:00 +02:00
Simon Frei
ec7c88ca55 lib/protocol: Fix yet another deadlock (fixes #5678) (#5679)
* lib/protocol: Fix yet another deadlock (fixes #5678)

* more consistency

* read deadlock

* naming

* more naming
2019-05-02 09:21:07 +01:00
Simon Frei
19b51c9b92 lib/protocol: Don't close asyncMessage.done on success (fixes #5674) (#5675) 2019-04-29 17:52:57 +02:00
Audrius Butkevicius
0ca1f26ff8
lib/versioner: Restore for all versioners, cross-device support (#5514)
* lib/versioner: Restore for all versioners, cross-device support

Fixes #4631
Fixes #4586
Fixes #1634
Fixes #5338
Fixes #5419
2019-04-28 23:30:16 +01:00
Jakob Borg
2984d40641 lib/ignore: Additional test case (#5672) 2019-04-28 21:20:11 +01:00
Simon Frei
5da41f75fa lib/model, lib/protocol: Wait for reader/writer loops on close (fixes #4170) (#5657)
* lib/protocol: Wait for reader/writer loops on close (fixes #4170)

* waitgroup

* lib/model: Don't hold lock while closing connection

* fix comments

* review (lock once, func argument) and naming
2019-04-28 11:58:51 +01:00
Simon Frei
04b927104f lib/protocol: Don't send any messages before cluster config (#5646)
* lib/model: Send cluster config before releasing pmut

* reshuffle

* add model.connReady to track cluster-config status

* Corrected comments/strings

* do it in protocol
2019-04-23 20:47:11 +01:00
Simon Frei
110806842c lib/model: Pass the old not new fileinfo to deleteItemOnDisk (fixes #5654) (#5655) 2019-04-23 20:46:28 +01:00
Simon Frei
e302ccf4b4
lib/fs: Improve IsParent (#5658) 2019-04-22 11:12:32 +02:00
Simon Frei
926e9228ed lib/scanner, lib/model: File -> item when logging error (#5664) 2019-04-21 16:19:59 +01:00
Simon Frei
86e72d9973 lib/model: Use RLock and legacy polish (#5663) 2019-04-21 13:21:36 +01:00
Simon Frei
3bea59b0d9
lib/model: Refactor progressEmitter to de-/activate by config (fixes #4613) (#5623) 2019-04-13 14:20:51 +02:00
Simon Frei
fca895a632 lib/model: Fix block index calculation for recheckFile (fixes #5649) (#5648) 2019-04-12 15:21:07 +02:00
Simon Frei
79360e2205
lib/fs: Add test that symlinks are skipped on walk (#5644) 2019-04-10 22:36:37 +02:00
Simon Frei
9b2a73f9ab
lib/model: Pause pull for at least as long as failed pull took (fixes #5641) (#5643) 2019-04-10 16:22:23 +02:00
Simon Frei
c305265c62 lib/model: Request errors conforming to BEP specs (#5642) 2019-04-10 11:47:24 +02:00
Simon Frei
ca823bd591
lib/fs: When watching remove \\?\ for drive letters (fixes #5578) (#5633) 2019-04-09 09:02:04 +02:00
Simon Frei
395e524e2d lib/model: Update db on scan/pull in folder (#5608) 2019-04-07 13:29:17 +02:00
Jakob Borg
f0f8bf7784 lib/config: Round times stored for pending folders/devices (fixes #5554) 2019-03-27 20:35:42 +01:00
Jakob Borg
bf3834e367
lib/protocol: Use constants instead of init time hashing (fixes #5624) (#5625)
This constructs the map of hashes of zero blocks from constants instead
of calculating it at startup time. A new test verifies that the map is
correct.
2019-03-27 20:20:30 +01:00
Simon Frei
8d1eff7e41 lib/model: Missing queue.Done and reset between pulls (fixes #5332) (#5626) 2019-03-27 20:19:35 +01:00
Simon Frei
0cff66fcbc lib/model: Optimize puller for meta only changes (#5622) 2019-03-27 08:36:58 +00:00
Simon Frei
43a5be1c4b lib/model: Send item finished even after deregistering (fixes #5362) (#5620) 2019-03-26 21:31:33 +01:00
Simon Frei
b50039a920 cmd/syncthing, lib/api: Separate api/gui into own package (ref #4085) (#5529)
* cmd/syncthing, lib/gui: Separate gui into own package (ref #4085)

* fix tests

* Don't use main as interface name (make old go happy)

* gui->api

* don't leak state via locations and use in-tree config

* let api (un-)subscribe to config

* interface naming and exporting

* lib/ur

* fix tests and lib/foldersummary

* shorter URVersion and ur debug fix

* review

* model.JsonCompletion(FolderCompletion) -> FolderCompletion.Map()

* rename debug facility https -> api

* folder summaries in model

* disassociate unrelated constants

* fix merge fail

* missing id assignement
2019-03-26 19:53:58 +00:00
Simon Frei
d4e81fff8a lib/model: Remove unnecessary arguments in pulling functions (#5619) 2019-03-25 12:59:22 +01:00
Simon Frei
e31a116e6e lib/model: Pass correct file info to deleteItemOnDisk (fixes #5616) (#5617) 2019-03-25 12:42:39 +01:00
Simon Frei
e7ae851900 lib/model: Debug and test fixes (#5613) 2019-03-22 14:43:47 +01:00
Simon Frei
1a6d023ba8 lib/model: Run recvonly tests in temporary dirs (#5587) 2019-03-20 17:38:03 +00:00
Simon Frei
50d8c43e7c lib/config: Set UseLargeBlocks to true by default (fixes #5599) (#5600) 2019-03-12 12:59:26 +00:00
Simon Frei
289a02e994 lib/model: Integrate stat refs in folder (#5596) 2019-03-11 16:57:21 +00:00
Simon Frei
445637ebec lib/model: Pass fset & ignores on folder creation (#5592) 2019-03-11 07:28:54 +01:00
Simon Frei
3f3d2c814b
lib/model: Remove unused code (#5591) 2019-03-10 17:05:39 +01:00
Simon Frei
189e44488e lib/model: Introduce must test utility (#5586)
* lib/model: Introduce must test utility

* nice
2019-03-09 18:45:36 +00:00
Simon Frei
27ff20faa3 lib/model: Introduce waitForState test utility (#5585)
* lib/model: Introduce waitForState test utility

* folder id as param to waitForState
2019-03-09 10:36:55 +00:00
Simon Frei
b1564e53e4 lib/model: Improve test utilities (#5584) 2019-03-08 20:29:09 +00:00
Simon Frei
3a75b63776
lib/model: Use temp dir from osutils in tests (#5581) 2019-03-07 16:34:41 +01:00
Simon Frei
8e238c8e48
lib/model: Check before replacing existing file on pull (fixes #5571) (#5567) 2019-03-07 15:15:14 +01:00
Jakob Borg
9da3273eb8 lib/model: Clarify fileInfoBatch.flushIfFull criteria 2019-03-05 21:34:04 +01:00
Simon Frei
bd37f6da17 lib/model: Optimize dbUpdaterRoutine (#5576) 2019-03-05 21:32:37 +01:00
Simon Frei
6940d79f5b lib/model: Use errors.Wrap for pull errors (#5563) 2019-03-04 13:01:52 +00:00
Simon Frei
0f80318ef6
lib/scanner: Consistenlty use CreateFileInfo and remove outdated comment (#5574) 2019-03-04 13:27:33 +01:00
Simon Frei
d6622b1f68 lib/model: setUp -> setup (#5573) 2019-03-04 12:27:10 +00:00
Simon Frei
43bcb3d5a5 lib/model: Refactor conflict name handling (#5572) 2019-03-04 12:20:40 +00:00
Jakob Borg
f24676ba5a
lib/tlsutil: Enable TLS 1.3 when available, on test builds (fixes #5065) (#5558)
* lib/tlsutil: Enable TLS 1.3 when available, on test builds (fixes #5065)

This enables TLS 1.3 negotiation on Go 1.12 by setting the GODEBUG
variable. For now, this just gets enabled on test versions (those with a
dash in the version number).

Users wishing to enable this on production builds can set GODEBUG
manually.

The string representation of connections now includes the TLS version
and cipher suite. This becomes part of the log output on connections.
That is, when talking to an old client:

    Established secure connection .../TLS1.2-TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256

and now potentially:

    Established secure connection .../TLS1.3-TLS_AES_128_GCM_SHA256

(The cipher suite was there previously in the log output, but not the
TLS version.)

I also added this info as a new Crypto() method on the connection, and
propagate this out to the API and GUI, where it can be seen in the
connection address hover (although with bad word wrapping sometimes).

* wip

* wip
2019-02-26 11:49:02 +01:00
Simon Frei
722b3fce6a all: Hide implementations behind interfaces for mocked testing (#5548)
* lib/model: Hide implementations behind interfaces for mocked testing

* review
2019-02-26 08:09:25 +00:00
Audrius Butkevicius
fafd30f804 lib/scanner: Use standard adler32 when we don't need rolling (#5556)
* lib/scanner: Use standard adler32 when we don't need rolling

Seems the rolling adler32 implementation is super slow when executed on large blocks, even tho I can't explain why.

BenchmarkFind1MFile-16    				     100	  18991667 ns/op	  55.21 MB/s	  398844 B/op	      20 allocs/op
BenchmarkBlock/adler32-131072/#00-16     		     200	   9726519 ns/op	1078.06 MB/s	 2654936 B/op	     163 allocs/op
BenchmarkBlock/bozo32-131072/#00-16      		      20	  73435540 ns/op	 142.79 MB/s	 2654928 B/op	     163 allocs/op
BenchmarkBlock/buzhash32-131072/#00-16   		      20	  61482005 ns/op	 170.55 MB/s	 2654928 B/op	     163 allocs/op
BenchmarkBlock/buzhash64-131072/#00-16   		      20	  61673660 ns/op	 170.02 MB/s	 2654928 B/op	     163 allocs/op
BenchmarkBlock/vanilla-adler32-131072/#00-16         	     300	   4377307 ns/op	2395.48 MB/s	 2654935 B/op	     163 allocs/op
BenchmarkBlock/adler32-16777216/#00-16               	       2	 544010100 ns/op	  19.27 MB/s	   65624 B/op	       5 allocs/op
BenchmarkBlock/bozo32-16777216/#00-16                	       1	4678108500 ns/op	   2.24 MB/s	51970144 B/op	      24 allocs/op
BenchmarkBlock/buzhash32-16777216/#00-16             	       1	3880370700 ns/op	   2.70 MB/s	51970144 B/op	      24 allocs/op
BenchmarkBlock/buzhash64-16777216/#00-16             	       1	3875911700 ns/op	   2.71 MB/s	51970144 B/op	      24 allocs/op
BenchmarkBlock/vanilla-adler32-16777216/#00-16       	     300	   4010279 ns/op	2614.72 MB/s	   65624 B/op	       5 allocs/op
BenchmarkRoll/adler32-131072/#00-16                  	    2000	    974279 ns/op	 134.53 MB/s	     270 B/op	       0 allocs/op
BenchmarkRoll/bozo32-131072/#00-16                   	    2000	    791770 ns/op	 165.54 MB/s	     270 B/op	       0 allocs/op
BenchmarkRoll/buzhash32-131072/#00-16                	    2000	    917409 ns/op	 142.87 MB/s	     270 B/op	       0 allocs/op
BenchmarkRoll/buzhash64-131072/#00-16                	    2000	    881125 ns/op	 148.76 MB/s	     270 B/op	       0 allocs/op
BenchmarkRoll/adler32-16777216/#00-16                	      10	 124000400 ns/op	 135.30 MB/s	 7548937 B/op	       0 allocs/op
BenchmarkRoll/bozo32-16777216/#00-16                 	      10	 118008080 ns/op	 142.17 MB/s	 7548928 B/op	       0 allocs/op
BenchmarkRoll/buzhash32-16777216/#00-16              	      10	 126794440 ns/op	 132.32 MB/s	 7548928 B/op	       0 allocs/op
BenchmarkRoll/buzhash64-16777216/#00-16              	      10	 126631960 ns/op	 132.49 MB/s	 7548928 B/op	       0 allocs/op

* Update benchmark_test.go

* gofmt

* fixup benchmark
2019-02-25 13:29:31 +04:00
Simon Frei
ad5a046843
lib/fs: Rename fsFile* to basicFile* (#5546) 2019-02-24 18:02:02 +01:00
Simon Frei
ca3ae64bbf lib/db: Flush batch based on size and refactor (fixes #5531) (#5536)
Flush the batch when exceeding a certain size, instead of when reaching a number
of batched operations.
Move batch to lowlevel to be able to use it in NamespacedKV.
Increase the leveldb memory buffer from 4 to 16 MiB.
2019-02-14 23:15:13 +00:00
Simon Frei
d5ff2c41dc all: Get rid of fatal logging (#5537)
* cleanup Fatal in lib/config/config.go

* cleanup Fatal in lib/config/folderconfiguration.go

* cleanup Fatal in lib/model/model.go

* cleanup Fatal in cmd/syncthing/monitor.go

* cleanup Fatal in cmd/syncthing/main.go

* cleanup Fatal in lib/api

* remove Fatal methods from logger

* lowercase in errors.Wrap

* one less channel
2019-02-14 20:29:14 +00:00
Simon Frei
905c3594b0 lib/model: Various model test fixes and polish (#5528)
* lib/model: Various model test fixes and polish

Missing calls to m.Stop()
Don't fail test if temporary test dir cleanup fails

* drop lazyness
2019-02-13 18:54:04 +00:00
Simon Frei
225c0dda80 lib/model: Scan conflicts after creation (#5511)
Also unflakes and improve TestRequestRemoteRenameChanged.
2019-02-12 16:05:20 +01:00
Simon Frei
5fd2cab102 lib/model: Run more tests in tmp dir (#5527) 2019-02-12 16:04:04 +01:00
Simon Frei
4299af1c63 lib/config, lib/model: Use path from locations to check disk space for db (#5525) 2019-02-12 12:25:11 +00:00
Simon Frei
d85ef949be lib/model: Introduce setupModel test utility (#5524) 2019-02-12 12:18:13 +00:00
Audrius Butkevicius
dc929946fe all: Use new reflect based CLI (#5487) 2019-02-12 07:58:24 +01:00
Simon Frei
7bac927ac8 lib/model: Use functions to generate config (#5513) 2019-02-12 07:50:07 +01:00
Jakob Borg
04fdafa280 lib/db, lib/model: Remove dead code (#5517) 2019-02-08 16:42:58 +01:00
Simon Frei
2f415d8f09 lib/model: Don't use LocalDeviceID as normal id in tests (#5512) 2019-02-06 09:32:03 +01:00
Simon Frei
e538797ce1 lib/model: Improve TestIssue5063 (#5509)
* lib/model: Improve TestIssue5063

* add not that helpful issue comment
2019-02-05 23:07:21 +00:00
Simon Frei
5df8219bcb lib/model: Add progressEmitter to supervisor (model) (#5510) 2019-02-05 18:02:36 +00:00
Simon Frei
af4fb97538 lib/model: Fail test instead of panic due to closing channel twice (#5508) 2019-02-05 18:01:56 +00:00
Simon Frei
5d9d87f770 lib/model: Helperize test os and remove error return value (#5507) 2019-02-05 18:01:05 +00:00
Simon Frei
82afe73a9a cmd/syncthing, lib/config: Update default config creation (#5492)
Also remove dead code in config.Wrapper.
2019-02-02 12:43:57 +01:00
Jakob Borg
c2ddc83509 all: Revert the underscore sillyness 2019-02-02 12:16:27 +01:00
Jakob Borg
9fd270d78e
all: A few more interesting linter fixes (#5502)
A couple of minor bugs and simplifications
2019-02-02 12:09:07 +01:00
Jakob Borg
0b2cabbc31
all: Even more boring linter fixes (#5501) 2019-02-02 11:45:17 +01:00
Jakob Borg
df5c1eaf01
all: Bunch of more linter fixes (#5500) 2019-02-02 11:02:28 +01:00
Jakob Borg
2111386ee4
all: Fix some linter errors (#5499)
I'm working through linter complaints, these are some fixes. Broad
categories:

1) Ignore errors where we can ignore errors: add "_ = ..." construct.
you can argue that this is annoying noise, but apart from silencing the
linter it *does* serve the purpose of highlighting that an error is
being ignored. I think this is OK, because the linter highlighted some
error cases I wasn't aware of (starting CPU profiles, for example).

2) Untyped constants where we though we had set the type.

3) A real bug where we ineffectually assigned to a shadowed err.

4) Some dead code removed.

There'll be more of these, because not all packages are fixed, but the
diff was already large enough.
2019-02-02 10:11:42 +01:00
Simon Frei
583172dc8d lib/db: Fix race in NamespacedKV (#5496) 2019-02-01 09:54:21 +01:00
Simon Frei
7236d56731
lib/model: In tests disable watching for changes by default (fixes #5246) (#5485) 2019-01-30 16:38:10 +01:00
Simon Frei
657be162dd test, lib/rc: Integration test fixes and polish (#5488) 2019-01-29 16:59:00 +01:00
Simon Frei
79d109a386 lib/config: Add omitempty to DeprecatedMinHomeDiskFreePct (fixes #5482) (#5484) 2019-01-28 11:46:28 +01:00
Jakob Borg
75dcff0a0e
all: Copy owner/group from parent (fixes #5445) (#5479)
This adds a folder option "CopyOwnershipFromParent" which, when set,
makes Syncthing attempt to retain the owner/group information when
syncing files. Specifically, at the finisher stage we look at the parent
dir to get owner/group and then attempt a Lchown call on the temp file.
For this to succeed Syncthing must be running with the appropriate
permissions. On Linux this is CAP_FOWNER, which can be granted by the
service manager on startup or set on the binary in the filesystem. Other
operating systems do other things, but often it's not required to run as
full "root". On Windows this patch does nothing - ownership works
differently there and is generally less of a deal, as permissions are
inherited as ACLs anyway.

There are unit tests on the Lchown functionality, which requires the
above permissions to run. There is also a unit test on the folder which
uses the fake filesystem and hence does not need special permissions.
2019-01-25 09:52:21 +01:00
Simon Frei
a45ba70467 lib/model: Improve errors while pulling (#5474) 2019-01-24 08:18:55 +01:00
Simon Frei
42bd42df5a lib/db: Do all update operations on a single item at once (#5441)
To do so the BlockMap struct has been removed. It behaves like any other prefixed
part of the database, but was not integrated in the recent keyer refactor. Now
the database is only flushed when files are in a consistent state.
2019-01-23 10:22:33 +01:00
Simon Frei
a371b15398 lib/db: Various polish (#5471)
naming: buf -> keyBuf
dedup: use getFileTrunc
manually inline insertFile
2019-01-20 10:24:39 +01:00
Simon Frei
00fa77dd47 lib/db: Consistent use of buffers (#5470) 2019-01-20 08:47:20 +01:00
Simon Frei
df4d754197 lib/db: Minor polish (#5469) 2019-01-19 20:26:46 +01:00
Simon Frei
f3d735c56a lib/db: Fix, optimize and extend benchmarks (#5467) 2019-01-19 20:24:44 +01:00
Simon Frei
1d99db9bc6 lib/db: Deduplicate getFile* code (#5468) 2019-01-19 20:21:58 +01:00
Audrius Butkevicius
96bd691f55 lib/fs: Skip some tests on OpenBSD (fixes #5077) (#5466) 2019-01-19 08:28:57 +01:00
Simon Frei
22e133cce6 lib/db: Deduplicate comparing old and new items (#5465) 2019-01-18 21:19:56 +00:00
Simon Frei
1f87b874af lib/db: Add "dirty" function terminology to getGlobal (ref #5462) (#5463) 2019-01-18 13:01:39 +01:00
Jakob Borg
1e69997ecd
lib/db: Fix iterating sequence index (fixes #5340) (#5462)
There was a problem in iterating the sequence index that could result
in missing updates. The issue is that while the index was (correctly)
iterated in a snapshot, the actual file infos were read dirty outside of
the snapshot. This fixes this by doing the reads inside the snapshot,
and also updates a couple of other places that did the same thing more
or less harmfully (I didn't investigate).

To avoid similar issues in the future I did some renaming of the
getFile* methods - the ones in a transaction are just getFile, while the
ones directly on the database are variants of getFileDirty to highlight
what's going on.
2019-01-18 11:34:18 +01:00
Jakob Borg
76af0cf07b
lib/protocol: Remove support for v0.13 hello messages (#5461) 2019-01-17 20:48:43 +01:00
Jakob Borg
b01edca420
all: Update protobuf package 1.0.0 -> 1.2.0 (#5452)
Also adds a few file global options to keep the generated code similar
to what we already had.
2019-01-14 11:53:36 +01:00
Simon Frei
c87411851d lib/protocol: Fix potential deadlock when closing connection (ref #5440) (#5442) 2019-01-14 08:32:37 +01:00
Simon Frei
51f65bd23a lib/model: Reset pull errors when succeeding (fix #5450) (#5451) 2019-01-14 08:30:52 +01:00
Jakob Borg
93ae9a1c2e lib/logger: Clean up LOGGER_DISCARD handing to unbreak tests
When using newLogger() with an io.Writer, that writer should in fact be
used...
2019-01-13 10:22:01 +01:00
Simon Frei
0b03b6a9ec lib/model: Improve filesystem operations during tests (fixes #5422)
* lib/fs, lib/model: Improve filesystem operations during tests (fixes #5422)

Introduces MustFilesystem that panics on errors and should be used for operations
during testing which must never fail.
Create temporary directories outside of testdata.

* don't do a filesystem, just a wrapper around os for testing

* fix copyright
2019-01-11 12:56:05 +00:00
Simon Frei
24ffd8be99 all: Send Close BEP msg on intentional disconnect (#5440)
This avoids waiting until next ping and timeout until the connection is actually
closed both by notifying the peer of the disconnect and by immediately closing
the local end of the connection after that. As a nice side effect, info level
logging about dropped connections now have the actual reason in it, not a generic
timeout error which looks like a real problem with the connection.
2019-01-09 17:31:09 +01:00
Jakob Borg
1e71b00936
lib/model: Sanitize paths used for auto accepted folders (fixes #5411) (#5435) 2019-01-05 18:10:02 +01:00
Jakob Borg
5503175854
lib/logger: Strip control characters from log output (fixes #5428) (#5434) 2019-01-05 11:31:02 +01:00
Simon Frei
158559023e lib/db: Fix sequence updating for remote invalid items (#5420)
* lib/db: Fix sequence updating for remote invalid items

* fix for the unit test introduced in the previous commit

* lib/db: Polish blockmap
2019-01-04 20:19:10 +01:00
Jakob Borg
04070b4848
lib/logger: Add test for stack level (#5430) 2019-01-04 17:51:58 +01:00
Simon Frei
47e08797cb lib/model: Don't pull if ignores failed to load and cleanup (#5418) 2019-01-01 10:17:14 +01:00
Simon Frei
643cfe2e98 lib/config: Revert #5415 (#5417)
This reverts commit 9d075781ad:
"cmd/syncthing: Improve messages when free space is running out (#5415)"
2018-12-30 21:57:41 +01:00
Simon Frei
8bb9878f26 lib/model: Check folder context before setting error state (#5416) 2018-12-30 21:56:16 +01:00
Maurizio Tomasi
9d075781ad cmd/syncthing: Improve messages when free space is running out (#5415) 2018-12-29 20:48:20 +01:00
Jakob Borg
41469c5393 lib/rc: Remove dead lock (fixes #5403) 2018-12-23 09:11:15 +01:00
Simon Frei
4783294a09 lib/model: Don't pass nil *Matcher to performFinish (fixes #5401) (#5402) 2018-12-22 21:58:17 +01:00
Simon Frei
99c9d65ddf lib/scanner: Check ignore patterns before reporting error (fixes #5397) (#5398) 2018-12-21 12:08:15 +01:00
Simon Frei
fc81e2b3d7 lib/model: Do folder watch operations under lock (fixes #5392) (#5395) 2018-12-21 12:06:21 +01:00
Simon Frei
2626143fc5 lib/rc: Fix hangups when Syncthing process ends unexpectedly (#5383) 2018-12-21 11:49:04 +01:00
Jakob Borg
944ddcf768
all: Become a Go module (fixes #5148) (#5384)
* go mod init; rm -rf vendor

* tweak proto files and generation

* go mod vendor

* clean up build.go

* protobuf literals in tests

* downgrade gogo/protobuf
2018-12-18 12:36:38 +01:00
Simon Frei
3cc8918eb4 lib/db: Defer unlock to avoid hangup on panic (#5388) 2018-12-17 14:59:09 +01:00
Simon Frei
c40c9a8d6a
lib/scanner: Don't report error on missing items (fixes #5385) (#5387) 2018-12-17 14:52:15 +01:00
Jakob Borg
abb3fb8a31
lib/events: Become a service (fixes #5372) (#5373)
Here the event Logger is rewritten as a service with a main loop instead
of mutexes. This loop has a select with essentially two legs: incoming
events, and subscription changes. When both are possible select will
chose one randomly, thus ensuring that in practice unsubscribes will
happen timely and not block the system.
2018-12-13 13:42:28 +01:00
Jakob Borg
fc860df514 lib/watchaggregator: Properly unsubscribe from events when stopping (ref #5372) (#5374) 2018-12-13 08:11:51 +01:00
Simon Frei
c934918347 lib/scanner: Fix empty path on normalization error (fixes #5369) (#5370) 2018-12-12 12:09:39 +01:00
Simon Frei
a09079ed25 all: Display list of locally changed items in UI (fixes #5336) (#5337) 2018-12-11 09:59:04 +01:00
Jakob Borg
1b59960ff9
lib/model: Attempt to unflake Deregister tests (fixes #5362) (#5366) 2018-12-11 09:42:03 +01:00
Jakob Borg
f04d054b5a lib/db: Document current keyspace 2018-12-10 09:55:21 +01:00
Audrius Butkevicius
ff2cde469e lib/model: Allow limiting number of concurrent scans (fixes #2760) (#4888) 2018-12-05 08:40:05 +01:00
Jakob Borg
0fe4c01a28
lib/db: Fix broken test on Windows (#5347) 2018-12-05 08:24:14 +01:00
Jakob Borg
e89c4c053a lib/db: Simple case sensitivity test 2018-11-26 17:58:00 +01:00
Simon Frei
2f9840ddae lib: Introduce fs.IsParent (fixes #5324) (#5326) 2018-11-22 11:16:45 +01:00
Jakob Borg
c0a26c918a
lib/scanner, vendor: Update github.com/chmduquesne/rollinghash (fixes #5334) (#5335)
Updates the package and fixes a test that depended on the old behavior
of Write() being equivalent to Reset()+Write() which is no longer the
case. The scanner already did resets after each block write, so this is
fine.
2018-11-22 08:50:06 +01:00
Simon Frei
db8777c29e lib/model: Add test for #5323 (#5325) 2018-11-13 08:36:16 +00:00
Simon Frei
33bed5b1ec lib/model: Don't compare permissions if IgnorePerms is true (fixes #5323) (#5322) 2018-11-13 08:54:49 +01:00
Simon Frei
4f27bdfc27 lib/model, lib/protocol: Handle request concurrency in model (#5216) 2018-11-13 08:53:55 +01:00
Simon Frei
603da2dce2 cmd/syncthing, lib/relay: Fixes regarding stopping of services (#5293) 2018-11-07 11:05:07 +01:00
Simon Frei
d510e3cca3 all: Display errors while scanning in web UI (fixes #4480) (#5215) 2018-11-07 11:04:41 +01:00
Simon Frei
53b0f36be6 lib/fs: Use os.FileMode.String for fs.FileMode (#5302) 2018-10-31 12:49:50 +01:00
Simon Frei
01ade9c8ae lib/connections: Don't panic on removed device (fixes #5299) (#5300) 2018-10-30 10:34:19 +01:00
Simon Frei
b1acc37c16 lib/db: Update local need on device removal (fixes #5294) (#5295) 2018-10-30 05:40:51 +01:00
Simon Frei
64a591610b lib/model: Check if files from queue are invalid (fixes #5291) (#5292) 2018-10-26 19:13:35 +01:00
Simon Frei
406bedf1e3
lib/logger: Disable debug flags without debugging (fixes #4780) (#5278) 2018-10-23 15:17:40 +02:00
Alexandre Viau
9745679c63 lib: chmod -x on progressemitter.go and errors.go (#5281) 2018-10-21 16:08:14 +01:00
Jakob Borg
8519a24ba6
cmd/*, lib/tlsutil: Refactor TLS stuff (fixes #5256) (#5276)
This changes the TLS and certificate handling in a few ways:

- We always use TLS 1.2, both for sync connections (as previously) and
  the GUI/REST/discovery stuff. This is a tightening of the requirements
  on the GUI. AS far as I can tell from caniusethis.com every browser from
  2013 and forward supports TLS 1.2, so I think we should be fine.

- We always greate ECDSA certificates. Previously we'd create
  ECDSA-with-RSA certificates for sync connections and pure RSA
  certificates for the web stuff. The new default is more modern and the
  same everywhere. These certificates are OK in TLS 1.2.

- We use the Go CPU detection stuff to choose the cipher suites to use,
  indirectly. The TLS package uses CPU capabilities probing to select
  either AES-GCM (fast if we have AES-NI) or ChaCha20 (faster if we
  don't). These CPU detection things aren't exported though, so the tlsutil
  package now does a quick TLS handshake with itself as part of init().
  If the chosen cipher suite was AES-GCM we prioritize that, otherwise we
  prefer ChaCha20. Some might call this ugly. I think it's awesome.
2018-10-21 14:17:50 +09:00
Simon Frei
089c283ca6 lib/config: Disable folder free disk check when configured (fixes #5267) (#5268) 2018-10-12 12:34:56 +01:00
Jakob Borg
7b0c49a1b6 cmd/stindex: Add index checking mode ("idxck") (#5262) 2018-10-11 20:48:39 +01:00
Simon Frei
0690fe7585 lib/model: Unnecessary return (#5264) 2018-10-11 15:08:37 +02:00
Jakob Borg
3bc918ff78 lib/db: Properly remove FileInfos when dropping folder (#5260) 2018-10-11 12:09:44 +02:00
Simon Frei
3e50edf46f lib/db: More info in sequence panic msg (#5261) 2018-10-11 12:05:57 +02:00
Simon Frei
1b10607def lib/model: Don't check folder health if there is nothing to pull (fixes #2497) (#5255) 2018-10-11 11:33:21 +02:00
Jakob Borg
9d7a811e72
lib/model: Don't flake out on shortcutting files (ref #5258, #5257, #5234) (#5259)
In a recent change (#5201) this return disappeared. The effect is that
we first shortcut the file and then also treat it normally. This results
in to database updates after each other, which are bound to end up in
the same batch. This means we remove one sequence entry and add two.

Not marking the issues as fixed, because I need to do more testing and
there are other discrepancies...
2018-10-11 11:07:52 +02:00
Simon Frei
d10773c311 lib/db, lib/model: Resolve identical recv only items (fixes #5130) (#5230) 2018-10-10 12:43:07 +02:00
Jakob Borg
caa2356409 lib/db: Rename things (ref #5198)
This renames a couple of files to better reflect their current contents,
and moves a type. No lines of code actually changed.
2018-10-10 11:48:21 +02:00
Simon Frei
523ac45456 lib/model: Treat failed rename like del&update (#5203) 2018-10-10 11:37:20 +02:00
Jakob Borg
b50d57b7fd
lib/db: Refactor: use a Lowlevel type underneath Instance (ref #5198) (#5212)
This adds a thin type that holds the state associated with the
leveldb.DB, leaving the huge Instance type more or less stateless. Also
moves some keying stuff into the DB package so that other packages need
not know the keying specifics.

(This does not, yet, fix the cmd/stindex program, in order to keep the
diff size down. Hence the keying constants are still exported.)
2018-10-10 11:34:24 +02:00
Jakob Borg
f12ca95af2 lib/model: Unflake TestFolderRestartZombies (fixes #5244) 2018-10-07 13:58:25 +02:00
Jakob Borg
f528923d1e lib/model, cmd/syncthing: Wait for folder restarts to complete (fixes #5233) (#5239)
* lib/model, cmd/syncthing: Wait for folder restarts to complete (fixes #5233)

This is the somewhat ugly - but on the other hand clear - fix for what
is really a somewhat thorny issue. To avoid zombie folder runners a new
mutex is introduced that protects the RestartFolder operation. I hate
adding more mutexes but the alternatives I can think of are worse.

The other part of it is that the POST /rest/system/config operation now
waits for the config commit to complete. The point of this is that until
the commit has completed we should not accept another config commit. If
we did, we could end up with two separate RestartFolders queued in the
background. While they are both correct, and will run without
interfering with each other, we can't guarantee the order in which they
will run. Thus it could happen that the newer config got committed
first, and the older config commited after that, leaving us with the
wrong config running.

* test

* wip

* hax

* hax

* unflake test

* per folder mutexes

* paranoia

* race
2018-10-05 09:26:25 +01:00
Simon Frei
c9d6366d75 lib/connections: Don't info log about LAN if there are no rate limits (#5242) 2018-10-05 08:22:47 +02:00
Simon Frei
714a47ffb0 lib/config: Add context to the home disk out of space error (#5241) 2018-10-05 08:21:39 +02:00
Jakob Borg
d27463268d lib/fs: Add fakefs (#5235)
* lib/fs: Add fakefs

This adds a new fake filesystem type. It's described rather extensively
in fakefs.go, but the main point is that it's for testing: when you want
to spin up a Syncthing and have a terabyte or two of random files that
can be synced somewhere, or an inifitely large filesystem to sync files
into.

It has pseudorandom properties such that data read from one fakefs can
be written into another fakefs and read back and it will look
consistent, without any of the data actually being stored.

To use:

    <folder id="default" path="whatever" ...>
        <filesystemType>fake</filesystemType>

This will create an empty fake filesystem. You can also specify that it
should be prefilled with files:

    <folder id="default" path="whatever?size=2000000" ...>
        <filesystemType>fake</filesystemType>

This will create a filesystem filled with 2TB of random data that can be
scanned and synced. There are more options, see fakefs.go.

Prefilled data is based on a deterministic seed, so you can index the
data and restart Syncthing and the index is still correct for all the
stored data.
2018-10-02 19:29:06 +01:00
Simon Frei
c2b0d309fb lib/model: Prevent repeat db update (#5231) 2018-09-27 07:41:40 +02:00
Simon Frei
cb0950b3fd
lib/db: Improve VersionList.String (#5229) 2018-09-26 23:30:22 +02:00
Simon Frei
03d0f0dc34 lib/fs: Try EvalSymlinks without '\\?\' prefix on failure (fixes #5226) (#5227) 2018-09-26 19:28:20 +01:00
Jakob Borg
5a69e85e80
cmd/syncthing: Listen on UNIX socket (fixes #3616) (#5210)
This adds the ability to listen on (only) a UNIX socket.
2018-09-21 14:28:57 +02:00
Simon Frei
4f4781d254
lib/scanner: Centralise protocol.FileInfo creation (#5202) 2018-09-18 15:34:17 +02:00
Jakob Borg
6a87aac84f
lib/db: Refactor key handling (ref #5198) (#5199)
This breaks out the key generation stuff into a separate type. It's
cleaner on its own, and it prepares for future stuff.
2018-09-18 10:41:06 +02:00
Simon Frei
272fb3b444
all: Adjust windows perms in fs package (#5200) 2018-09-16 16:09:56 +02:00
Simon Frei
60eb9088ff lib/model: Extend shortcutFile (#5201) 2018-09-16 10:29:06 +01:00
Simon Frei
c8652222ef all: Check files on disk/in db when deleting/renaming (fixes #5194) (#5195) 2018-09-16 09:48:14 +02:00
Audrius Butkevicius
710dba7f84 gui: Add arrays for pending/ignored folders (fixes #5190) (#5192) 2018-09-13 22:52:16 +02:00
Simon Frei
a57fa9cfab lib/model: Polish (#5189) 2018-09-13 18:17:13 +02:00
Simon Frei
49d5eae66a lib/config: Actually modify config element (fixes #5185) (#5186) 2018-09-12 12:16:52 +01:00
Boris Rybalkin
1b1741de64 cmd/syncthing: Add LDAP authentication for GUI (fixes #5163) (#5169) 2018-09-11 23:25:24 +02:00
Simon Frei
50ba0fd079 lib/fs: Case insensitive conversion to rel path on windows (fixes #5183) (#5176) 2018-09-11 22:30:32 +02:00
Jakob Borg
9e00b619ab all, vendor: Switch back to non-forked thejerf/suture (#5171) 2018-09-08 12:56:56 +03:00
Jakob Borg
4cb6bb6f64 Merge branch 'release'
* release:
  lib/db: Fix inconsistency in sequence index (fixes #5149) (#5158)
2018-09-02 21:05:53 +02:00
Jakob Borg
b80da29b23 lib/db: Fix inconsistency in sequence index (fixes #5149) (#5158)
The problem here is that we would update the sequence index before
updating the FileInfos, which would result in a high sequence number
pointing to a low-sequence FileInfo. The index sender would pick up the
high sequence number, send the old file, and think everything was good.
On the receiving side the old file is a no-op and ignored. The file
remains out of sync until another update for it happens.

This fixes that by correcting the order of operations in the database
update: first we remove old sequence index entries, then we update the
FileInfos (which now don't have anything pointing to them) and then we
add the sequence indexes (which the index sender can see).

The other option is to add "proper" transactions where required at the
database layer. I actually have a branch for that, but it's literally
thousands of lines of diff and I'm putting that off for another day as
this solves the problem...
2018-09-02 21:02:28 +02:00
Jakob Borg
836ca50570
lib/db: Fix inconsistency in sequence index (fixes #5149) (#5158)
The problem here is that we would update the sequence index before
updating the FileInfos, which would result in a high sequence number
pointing to a low-sequence FileInfo. The index sender would pick up the
high sequence number, send the old file, and think everything was good.
On the receiving side the old file is a no-op and ignored. The file
remains out of sync until another update for it happens.

This fixes that by correcting the order of operations in the database
update: first we remove old sequence index entries, then we update the
FileInfos (which now don't have anything pointing to them) and then we
add the sequence indexes (which the index sender can see).

The other option is to add "proper" transactions where required at the
database layer. I actually have a branch for that, but it's literally
thousands of lines of diff and I'm putting that off for another day as
this solves the problem...
2018-09-02 20:58:32 +02:00
Simon Frei
c62ce007ea lib/fs: Don't add path separators at end of path (fixes #5144) (#5146) 2018-08-28 08:18:55 +02:00
Audrius Butkevicius
aec66045ef
lib/config: Rewrite pending notifications (fixes #2291) 2018-08-25 11:36:10 +01:00
Simon Frei
03c0537340 lib/model: Fix regressions detecting deletes/ignores (fixes #5125, fixes #5127) (#5129) 2018-08-25 10:32:35 +02:00
Jakob Borg
7dde6c7e3c lib/model: Clear out-of-space-errored files from queue (fixes #5143) 2018-08-25 10:26:10 +02:00
Simon Frei
cb0f4ce55a lib/model: Don't stop folder if out of disk space (fixes #2370) (#5099)
This removes the out of disk space check from CheckHealth. The disk space is now
only checked if there are files to pull, in which case pulling those files is
stopped, but everything else (dirs, links, deletes) keeps running -> can recover
disk space through pulling.
2018-08-25 10:16:38 +02:00
Jakob Borg
a02db70a63 lib/model: Process download progress messages for all folder types (fixes #5131) (#5139) 2018-08-21 18:49:35 +01:00
Simon Frei
165417c462 lib/model: Fixes on receive-only test setup and pulling (#5136) 2018-08-19 22:34:26 +01:00
Simon Frei
ff3cbdc90d lib/model: Check availability later to catch renames (#5097) 2018-08-19 19:03:20 +01:00
Simon Frei
9028969617
lib/model: Small fixes to test convenience functions (#5128) 2018-08-16 12:11:48 +02:00
Simon Frei
7c0798b622 lib/model: Always release the lock (#5126) 2018-08-15 16:44:59 +02:00
Simon Frei
ee42c46bd3 lib/model: Catch racy nil deref in ClusterConfig (#5106) 2018-08-15 16:44:20 +02:00
Simon Frei
885f6fcf28 lib/model: Always release the lock (#5126) 2018-08-15 16:33:03 +02:00
Simon Frei
dc57bab107 lib/model: Don't enter pulling state if we need nothing (fixes #4782) (#5118) 2018-08-13 20:39:25 +02:00
Jakob Borg
48795dba07
all: Don't let Suture capture panics (fixes #4758) (#5119)
Fork with new option.
2018-08-13 20:39:08 +02:00
Simon Frei
c55c0c8c28
lib/watchaggregator: Don't delay mixed events only (#5094)
Also fix a minor bug in testing failure output.
2018-08-13 09:14:03 +02:00
Simon Frei
e20679afe1 lib/fs: Evaluate root when watching not on fs creation (fixes #5043) (#5105) 2018-08-11 22:24:36 +02:00
Jakob Borg
b37c05c6b8
lib/model: Don't run watcher on recvonly tests (fixes #5110) (#5112) 2018-08-11 22:19:37 +02:00
Simon Frei
dfe4008607 lib/model: Catch racy nil deref in ClusterConfig (#5106) 2018-08-11 09:10:29 +02:00
Jakob Borg
5cb4a9acf6 lib/db: Don't account remote invalid files (fixes #5089) (#5090) 2018-07-31 13:00:03 +02:00
Oyebanji Jacob Mayowa
adc5bf6604 lib/upnp: Don’t log unknown device types (fixes #5038) (#5087) 2018-07-30 16:34:35 +02:00
Audrius Butkevicius
5161f03f02 lib/config: Fix aliased append, copy config inputs and outputs (fixes #5063) (#5069) 2018-07-26 23:14:12 +02:00
Jakob Borg
3d8344003e lib/logger: Add missing dots (fixes #5073) 2018-07-19 20:49:57 +02:00
Audrius Butkevicius
9f87fd1fcf lib/model: More auto accept tests (#5014) 2018-07-15 20:26:20 +03:00
Simon Frei
5592b8b190 lib/model: Record error for unavailable files (#5066) 2018-07-14 14:09:23 +01:00
Jakob Borg
f822b10550
all: Add receive only folder type (#5027)
Adds a receive only folder type that does not send changes, and where the user can optionally revert local changes. Also changes some of the icons to make the three folder types distinguishable.
2018-07-12 11:15:57 +03:00
Simon Frei
6b82538e62 lib/model: Also handle missing parent dir non-regular items (#5048)
This is an improvement of PR #4493 and related to (and maybe fixing) #4961
and #4475. Maybe fixing, because there is no clear reproducer for that
problem.

The previous PR added a mechanism to resurrect missing parent directories,
if there is a valid child file to be pulled. The same mechanism does not
exist for dirs and symlinks, even though a missing parent can happen for
those items as well. Therefore this PR extends the resurrection to all types
of pulled items.

In addition I moved the IsDeleted branch while iterating over
processDirectly to the existing IsDeleted branch in the WithNeed iteration.
This saves one pointless assignment and IsDeleted query. Also
2018-07-10 18:40:06 +03:00
Simon Frei
3f17bda786 lib/db: Catch unignored/conflicting files as needed (fixes #5053) (#5054) 2018-07-10 18:32:34 +03:00
Simon Frei
409cb2beb8 lib/fs: Catch size-preserving changes on windows (fixes #5050) (#5056) 2018-07-09 18:29:22 +01:00
Simon Frei
0f0290d574 lib/model, lib/weakhash: Abort pulling quicker on folder stop (ref #5028) 2018-07-04 08:07:33 +01:00
Jakob Borg
ff441d3b3e
lib/connections: Don't spin on accept failures (fixes #5025) (#5036) 2018-06-27 08:24:30 +02:00
Simon Frei
881e923105 cmd/syncthing, lib/db: Abort execution if db version is too high (fixes #4994) (#5022) 2018-06-26 11:40:34 +02:00
Jakob Borg
ef5ca0c218 build: Let "go generate" create assets 2018-06-26 10:29:36 +02:00
Simon Frei
7b0d8c2e77 lib/model: Release both locks when waiting for services to stop (fixes #5028) 2018-06-24 16:55:28 +01:00
Jakob Borg
b1b68ceedb
Add LocalFlags to FileInfo (#4952)
We have the invalid bit to indicate that a file isn't good. That's enough for remote devices. For ourselves, it would be good to know sometimes why the file isn't good - because it's an unsupported type, because it matches an ignore pattern, or because we detected the data is bad and we need to rescan it.

Or, and this is the main future reason for the PR, because it's a change detected on a receive only device. We will want something like the invalid flag for those changes, but marking them as invalid today means the scanner will rehash them. Hence something more fine grained is required.

This introduces a LocalFlags fields to the FileInfo where we can stash things that we care about locally. For example,

    FlagLocalUnsupported = 1 << 0 // The kind is unsupported, e.g. symlinks on Windows
    FlagLocalIgnored     = 1 << 1 // Matches local ignore patterns
    FlagLocalMustRescan  = 1 << 2 // Doesn't match content on disk, must be rechecked fully

The LocalFlags fields isn't sent over the wire; instead the Invalid attribute is calculated based on the flags at index sending time. It's on the FileInfo anyway because that's what we serialize to database etc.

The actual Invalid flag should after this just be considered when building the global state and figuring out availability for remote devices. It is not used for local file index entries.
2018-06-24 09:50:18 +02:00
Simon Frei
c2784d76e4 lib/db: Remove updated invalid files from need bucket (fixes #5007) (#5008) 2018-06-18 08:23:40 +02:00
Simon Frei
8ff7ceeddc lib/ignore, lib/scanner: Fix recursion to catch included paths (fixes #5009) (#5010) 2018-06-18 08:22:19 +02:00
Simon Frei
54e17c8bf4 lib/model: Don't set watch error on folder creation (fixes 5005) (#5006) 2018-06-15 23:33:23 +01:00
Jakob Borg
35a75a95dc lib/model: Don't panic when rechecking file (fixes #5002) (#5003) 2018-06-13 18:07:52 +01:00
Simon Frei
9e0e04f4de all: Fix FS watcher restarting and web UI indication (fixes #4923) (#4962) 2018-06-11 15:47:54 +02:00
Simon Frei
1e2732aa21
lib/config, lib/model: Don't warn and return error (#4997) 2018-06-10 15:41:20 +02:00
Simon Frei
b7234785f8
lib/model: Wait for folder to stop (fixes #4981) (#4982) 2018-06-10 13:24:59 +02:00
Simon Frei
30056cd1ae lib/db: Move database schema migration into its own file (#4985) 2018-06-08 12:46:00 +02:00
Jakob Borg
a467f6908c
lib/protocol: Test for IsEquivalent (#4996) 2018-06-08 12:02:16 +02:00
Simon Frei
e2c44f519c
lib/config, lib/model: Handle shared with information in config (fixes #4870) (#4974) 2018-06-06 23:34:11 +02:00
Jakob Borg
76f9e5c5db
lib/protocol: Correct block size calculation on 32 bit archs (fixes #4990) (#4991) 2018-06-06 09:59:33 +02:00
Jakob Borg
27d675a793 lib/upgrade: Tests should pass on darwin-386 2018-06-06 09:47:13 +02:00
Simon Frei
ee6516aa31
lib/fs: Resolve 8.3 filenames from watcher (ref #3800) (#4975) 2018-06-04 13:41:03 +02:00
Jakob Borg
8b15624f7d lib/scanner: Skip block size hysteresis test in -short mode
This burns like a percent of my laptop battery on every invocation...
2018-06-02 13:10:05 +00:00
Simon Frei
5baa432906 lib/db: Add index to track locally needed files (#4958)
To optimize WithNeed, which is called for the local device whenever an index
update is received. No tracking for remote devices to conserve db space, as
WithNeed is only queried for completion.
2018-06-02 15:08:32 +02:00
Audrius Butkevicius
d60f0e734c lib/scanner: Copy execute bits from previous version on Windows (fixes #4969) (#4970) 2018-05-29 07:01:23 +01:00
Simon Frei
a83176c77a lib/watchaggregator: Speedup propagation of removals (fixes #4953) (#4955) 2018-05-26 10:08:23 +01:00
Simon Frei
07bf24a3b4 lib/watchaggregator: Prevent race on config update (#4938) 2018-05-24 19:47:15 +01:00
Simon Frei
ef1633ac76
lib/db: Update global count when removing the previous global version (#4968) 2018-05-24 18:17:45 +02:00
Simon Frei
9d2b744c12 lib/model: Move String method to folder (#4964) 2018-05-23 08:23:21 +01:00
Jakob Borg
119335930c lib/model: Refactor override implementation into sendOnlyFolder
I'm trying to slowly clean this up a bit, and moving functionality out
into the folder types and having those methods not reach into model is
part of it. That can mean takign some odd arguments in the meantime,
some of those should probably become interfaces or properties on folder
in the long term.
2018-05-21 09:09:47 +02:00
Jakob Borg
370a3549e7 lib/model: Refactor folderScanner into folder
The functionality was anyway mostly implemented there and not isolated
in the folderScanner type. The attempt to refactor it out in the other
direction wouldn't work given that the event loop and stuff is on
`folder`.
2018-05-21 08:47:58 +02:00
Simon Frei
d64d954721 lib/db: Fix prefixed walks (fixes #4925) (#4940) 2018-05-17 09:26:40 +02:00
Jakob Borg
c08024ebb8
lib/db: Add db and test with invalid files (#4954)
This adds a couple of utilities for transporting databases in JSON and a
test to load a database and verify a couple of invalid bits. The test
itself is quite pointless at the moment, but it lays the groundwork for
testing the migration of this data in the next step (after the invalid
bit should be changed to local flags for local files).

When that happens we need to have a database in the old format already
there in order to be able to test the migration.
2018-05-16 08:44:08 +02:00
Simon Frei
eabce48761 lib/model: Also wait for ItemFinished in TestDeregister (#4951) 2018-05-14 20:01:35 +01:00
Simon Frei
d59aecba31
lib/ignore, lib/scanner: Catch included items below ignored ones (#4811) 2018-05-14 09:47:23 +02:00
Jakob Borg
20aa53486a
all: Serialize folder types to new names (#4942)
It's been a year and a half since we started accepting the new names.
It's time we start producing them.
2018-05-13 09:58:00 +02:00
Simon Frei
a7e30c925f all: Use Executable from os instead of osext (fixes #4900) (#4950) 2018-05-13 07:43:24 +02:00
Jakob Borg
9e696a154b lib/model: Rename {ro,rw}folder.go
To newer names better reflecting their types and yet sorting together
with folder.go. Doing it now without asking because there are no open
PRs that will get killed by it, and to avoid bikeshedding the names.
2018-05-11 11:00:02 +02:00
Jakob Borg
344e52e311
lib/model: Refactor {ro,rw}folder serve loop into just folder (#4939)
The actual pull method (which is really the only thing that differs
between them) is now an interface member which gets overridden by the
subclass.

"Subclass?!" Well, this is dynamic dispatch with overriding, I guess.
2018-05-11 10:45:13 +02:00
Simon Frei
a2f51c85c2 lib/osutil: Add test for IsDeleted (ref #4925) (#4936) 2018-05-10 21:39:33 +02:00
Jakob Borg
fb198a0645 lib/db: Actually delete the correct sequence prefix 2018-05-09 12:06:29 +02:00
xjtdy888
506181599c lib/db: Remove all sequences related to the folder (fixes #4928) (#4929) 2018-05-09 08:57:42 +02:00
Audrius Butkevicius
a48a31e3f5 lib/ignores: Fix ignore loading, report errors to UI (fixes #4901) (#4932) 2018-05-08 23:37:13 +02:00
Jakob Borg
2343c82c33 lib/model: Don't include unshared folders in ClusterConfig (fixes #4926)
Also fixes a data race where ClusterConfig would access folderFiles
without a lock. Tweaked the ClusterConfig unit test to verify the
behavior.
2018-05-08 08:19:34 +01:00
Simon Frei
a94aceb22f lib/model: Don't create folder root when paused (fixes #4903) (#4904) 2018-05-05 10:30:39 +02:00
Audrius Butkevicius
ef0dcea6a4 lib/model: Verify request content against weak (and possibly strong) hash (#4767) 2018-05-05 10:24:44 +02:00
Simon Frei
53dc346583 lib/model: Fix test function for introducer (#4898) 2018-05-01 22:56:20 +01:00
Jakob Borg
c2f498fc82 lib/model: Units are hard (fixes #4910) 2018-05-01 22:50:23 +01:00
Simon Frei
a548014755 lib/db, lib/model: Add sequence->deviceKey to db for sending indexes (#4906)
Instead of walking and unmarshalling the entire db and sorting the resulting
file infos by sequence, add store device keys by sequence number in the
database. Thus only the required file infos need be unmarshalled and are already
sorted by index.
2018-05-01 23:39:15 +02:00
Simon Frei
f6458d1b8f lib/config, lib/model: Include paused folders in cluster config (fixes #4897) 2018-04-22 17:01:52 +01:00
Iain Barnett
4d3b5348ae lib/protocol: Add note about non-standard Luhn calculation (#4895)
Skip-check: authors
2018-04-20 18:52:03 +02:00
Simon Frei
eca076cf7d
lib/osutil: Fix TraversesSymlink with symlinked fs root on windows (fixes #4875) (#4886) 2018-04-17 22:53:06 +02:00
Jakob Borg
dbcf7a02a0 lib/model: Increase the default pull limit (fixes #4883)
Bumping the limit to 2 * the max block size (16 MiB) is a slight
increase compared to previously. Nonetheless I think it's good to allow
us to queue one request and have one on the way in, or conversely have
one large block on the way in and be able to ask for smaller blocks from
others at the same time.
2018-04-17 07:55:49 +01:00
Jakob Borg
19c7cd99f5 all: Implement variable sized blocks (fixes #4807) 2018-04-16 19:08:50 +01:00
Simon Frei
01aef75c96 lib/fs: Fix watcher panic due to casing on windows (fixes #4877) (#4878) 2018-04-16 20:07:00 +02:00
Simon Frei
4072ae4d05 lib/model: Prevent warning on request in paused folder (fixes #4870) 2018-04-09 20:55:52 +01:00
Simon Frei
26d87ec3bb lib/fs: Don't panic when watching a folder with symlinked root (#4846) 2018-03-28 22:01:25 +01:00
Audrius Butkevicius
720e8dedbc
lib/osutil: Use unix lowprio implementation on Android (#4844) 2018-03-27 22:03:09 +01:00
Simon Frei
69f2c26d50 lib/scanner, lib/model: Actually assign version when un-ignoring (fixes #4841) (#4842)
This fixes a mistake introduced in #4750 and #4776 and is relevant to v0.14.46-rc1
2018-03-27 16:24:20 -04:00
Jakob Borg
bdbaa84989 lib/connections: Wrong context snuck in somehow 2018-03-27 07:18:26 -04:00
Jakob Borg
c49d864f14 lib/connections: Slightly refactor limiter juggling
Two small behavior changes: don't "charge" the data to the global rate
limit until it's been accepted by the device specific limiter, and fix
the send/recv direction in the log print on per device rate limits.
2018-03-26 11:45:02 -04:00
qepasa
2621c6fd2f lib/connections, lib/config: Bandwidth throttling per remote device (fixes #4516) (#4603) 2018-03-26 12:01:59 +02:00
Audrius Butkevicius
a5e40563de lib/model: Don't log ignored files (fixes #4832) (#4837) 2018-03-25 22:12:50 +02:00
Simon Frei
a557d62c4a all: Transition to using fs watcher by default (fixes #4552) 2018-03-25 21:05:47 +01:00
Simon Frei
da3b38ccce lib/fs: Fix and update error about inotify watch limit (fixes #4833) (#4835) 2018-03-23 12:56:38 +01:00
Graham Miln
e7dc2f9190 lib/nat: Fix clearAddresses/notify deadlock (ref #4601) (#4829)
clearAddresses write locks the struct and then calls notify. notify in turn tries to obtain a read lock on the same mutex. The result was a deadlock. This change unlocks the struct before calling notify.
2018-03-21 08:02:32 +01:00
Simon Frei
81bd428b25 review & forgotten fixes from other PR 2018-03-18 01:10:13 +00:00
Simon Frei
5520022766 lig/ignore, lib/logger: Fix race and potential dead-locks when logging (#4821) 2018-03-17 16:49:12 +01:00
Jakob Borg
bea3c01772 vendor: github.com/Zillode/notify is now github.com/syncthing/notify (#4813)
Given that we've taken on the resposibility of maintaining this forked
package I've added it to the Syncthing organization. We still vendor it
like an external package, because it's convenient to keep it as a fork
of upstream to easier merge and file pull requests towards them.
2018-03-14 14:48:22 +01:00
Simon Frei
55a7830ff9 lib/fs, lib/model: Make tests caching compatible (fixes #4749) (#4804) 2018-03-13 14:03:10 +01:00
Simon Frei
b31bad1c4d lib/model: Remove unused shouldIgnore function (#4805) 2018-03-12 20:16:40 +01:00
Simon Frei
8b4346c3ec lib/scanner, lib/fs: Don't create file infos with abs paths (fixes #4799) (#4800) 2018-03-12 13:18:59 +01:00
Jakob Borg
1471c15b29
cmd/syncthing, lib/db: Be nicer about dropping deltas on upgrade (#4798)
When dropping delta index IDs due to upgrade, only drop our local one.
Previously, when dropping all of them, we would trigger a full send in
both directions on first connect after upgrade. Then the other side
would upgrade, doing the same thing. Net effect is full index data gets
sent twice in both directions.

With this change we just drop our local ID, meaning we will send our
full index on first connect after upgrade. When the other side upgrades,
they will do the same. This is a bit less cruel.
2018-03-10 11:42:01 +01:00
Jakob Borg
d4b7be009c cmd/syncthing: Reset delta indexes on upgrade 2018-02-26 22:22:19 +00:00
Audrius Butkevicius
2751be57dc lib/connections: Fix relay connections when two devices use the same relay (fixes #4778) (#4779) 2018-02-25 16:12:46 +01:00
Audrius Butkevicius
8df90bb475 lib/scanner: Track modified by in symlinks (#4777) 2018-02-25 14:51:37 +01:00
Simon Frei
36251b86f7 lib/model: Mark deleted file as conflicting when un-ignoring (#4776)
This completes #4750 as a followup to #4765.
2018-02-25 13:03:55 +01:00
Jakob Borg
42cc64e2ed
lib/config, lib/model: Auto adjust pullers based on desired amount of pending data (#4748)
This makes the number of pullers vary with the desired amount of outstanding requests instead of being a fixed number.
2018-02-25 10:14:02 +01:00
Simon Frei
158859a1e2 lib: Handle metadata changes for send-only folders (fixes #4616, fixes #4627) (#4750)
Unignored files are marked as conflicting while scanning, which is then resolved
in the subsequent pull. Automatically reconciles needed items on send-only
folders, if they do not actually differ except for internal metadata.
2018-02-25 09:39:00 +01:00
Simon Frei
5822222c74 lib/model: More precise deletion detection (fixes #2571, fixes #4573) (#4765) 2018-02-25 09:27:54 +01:00
Jakob Borg
e99be02055 lib/model: Don't panic if block size in index is larger than protocol block size
This doesn't happen today, but it might in the future if the block size
were increased or made variable and we were talking to a client from the
future.
2018-02-24 07:34:23 -05:00
Matic Potočnik
1901a5a9f4 all: Fix typos (#4772)
Skip-check: authors
2018-02-24 08:51:29 +01:00
Jakob Borg
c9ec6159e8 lib/fs, vendor: s/zillode/Zillode/ 2018-02-21 08:27:33 +01:00
Simon Frei
6802505dda lib/fs: Fix MkdirAll failure due to \\?\ (fixes #4762) (#4763) 2018-02-16 15:19:20 +01:00
Jakob Borg
7a92f6c6b1
lib/db: Don't panic on negative counts (#4761)
* lib/db: Don't panic on negative counts (fixes #4659)

So, negative counts should never happen and hence the original idea to
panic. However, this sucks as the panic will happen in a folder runner,
be automatically swallowed by suture, and the runner gets restarted but
now we are in a bad state. (Related: #4758)

At the time of writing the global list is somewhat in flux (we've
changed how ignored files are handled, invalid bits, etc.) and I think
that can cause unusual conditions here. Hence just fixing up the numbers
instead until the next full recount.
2018-02-14 11:25:34 +01:00
Simon Frei
68c1b2dd47 all: Revert simultaneously walk fs and db on scan (fixes #4756) (#4757)
This reverts commit 6d3f9d5154.
2018-02-14 08:59:46 +01:00
Jakob Borg
c120c3a403 lib/scanner: Error handling in walk function (fixes #4753) (#4754) 2018-02-13 10:02:07 +00:00
Simon Frei
4955297bf6
lib/protocol: Invalid files should always lose (#4747) 2018-02-10 19:40:57 +01:00
Simon Frei
6d3f9d5154
all: Simultaneously walk fs and db on scan (fixes #2571, fixes #4573) (#4584)
When scanner.Walk detects a change, it now returns the new file info as well as the old file info. It also finds deleted and ignored files while scanning.
Also directory deletions are now always committed to db after their children to prevent temporary failure on remote due to non-empty directory.
2018-02-10 16:56:53 +01:00
Jakob Borg
b97d5bcca8
Remove KCP (fixes #4737) (#4741) 2018-02-09 11:40:57 +01:00
Kropekk
bc7639b0ff lib/versioner: Fix external versioner command specification on Windows (fixes #4560) 2018-02-07 14:12:27 +00:00
Simon Frei
8a3e584c19 lib/fs: Introduce walkfs debug facility (#4712) 2018-02-05 11:07:56 +01:00
Simon Frei
f87f13081b all: Display fs watcher status and retry starting it (ref #4552) (#4713) 2018-02-04 22:46:24 +01:00
Jakob Borg
98f2875b22
lib/fs: Further unflake watch tests (#4735) 2018-02-04 22:25:59 +01:00
Simon Frei
14a5561e43 lib/db: Fix benchmarks
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4708
2018-01-28 11:26:01 +00:00
Jakob Borg
0fe3ae7c22 lib/fs: Unflake watch tests (fixes #4687)
This removes a number of timing related things, leaving just the total
test timeout now bumped to one minute. Normally we get the filesystem
events within a second or so, so this doesn't affect the test time in
the successfull case. If we don't actually get the events we expect
within a minute I think we are legitimately in "failed" territory.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4715
LGTM: imsodin, AudriusButkevicius
2018-01-28 10:44:43 +00:00
Jakob Borg
5d0eb80204 lib/protocol: Disable broken KCP benchmark 2018-01-28 10:41:03 +01:00
Simon Frei
80079e8322 lib/fs: Use correct facility name
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4710
2018-01-27 12:52:48 +00:00
Simon Frei
364f61bda6 lib/db: Update global counts on invalidation (fixes #4701)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4702
2018-01-27 09:09:13 +00:00
Jakob Borg
050f9f8091 all: Mac OS X is now called macOS
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4694
LGTM: imsodin
2018-01-27 09:07:19 +00:00
Jakob Borg
e0931e201e lib/config: Reject empty folder IDs
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4698
LGTM: imsodin
2018-01-27 09:06:37 +00:00
Jakob Borg
4afe0f407a lib/config: Verify that we reject invalid device IDs when deserializing JSON 2018-01-24 09:25:41 +01:00
Nicholas Rishel
a505231774 lib/scanner: Support walking a symlink root (ref #4353)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4666
LGTM: AudriusButkevicius, imsodin
2018-01-24 00:05:47 +00:00
Simon Frei
885e3f19bd lib/ignores: Update lines even if patterns didn't change (fixes #4689)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4690
2018-01-20 07:52:57 +00:00
Simon Frei
93b5180e62 lib/model: Refactor Index/IndexUpdate
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4688
LGTM: calmh, AudriusButkevicius
2018-01-19 14:33:16 +00:00
Audrius Butkevicius
8e9c9b9553 lib/osutil: Fix priority lowering on Windows
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4686
2018-01-18 17:03:24 +00:00
Simon Frei
fae2ca8458 lib/db: Do not modify underlying array of argument
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4685
2018-01-18 12:40:43 +00:00
Jakob Borg
3af4164c8b lib/osutil: Don't attempt to reduce our niceness level (fixes #4681)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4682
2018-01-18 08:34:56 +00:00
Simon Frei
a1761795fe lib/ignore: Only handle lines prefixed with #include specially (fixes #4680)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4684
LGTM: AudriusButkevicius, calmh
2018-01-17 16:56:53 +00:00
Jakob Borg
e147db5233 lib/osutil: Check PGID before trying to set it (fixes #4679)
Fixes "permission denied" return when are already process group /
session leader.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4678
2018-01-17 12:32:11 +00:00
Jakob Borg
582539a1e6 lib/osutil: Disable setting priority on Windows (fixes #4676)
Presumably fixing the crash, leaving us to improve this calmly.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4677
2018-01-17 10:14:36 +00:00
Jakob Borg
c554ffccc9 cmd/syncthing, lib/config, lib/osutil: Lower process priority (fixes #4628)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4675
2018-01-15 17:11:14 +00:00
Jakob Borg
838c182b5b cmd/syncthing, lib/sync: Don't do deadlock detection when STDEADLOCKTIMEOUT=0 (fixes #4644)
Allows setting STDEADLOCKTIMEOUT=0 (or any integer <= 0) to disable the
deadlock detection entirely.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4673
2018-01-15 13:33:52 +00:00
Jakob Borg
fcc6a677a5 lib/upgrade: Always return latest version, even if older than current (fixes #4654)
The only special check remaining is the one to prefer a minor upgrade
over a major one.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4672
2018-01-15 12:13:25 +00:00
Simon Frei
bd63fd73b1 lib/model: Microoptimization of unifySubs and blockDiff
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4671
LGTM: AudriusButkevicius, calmh
2018-01-14 21:52:41 +00:00
Simon Frei
fecb21cdb1 gui: New rest endpoint to get errors when web UI is opened
Since #4340 pulls aren't happening every 10s anymore and may be delayed up to 1h.
This means that no folder error event reaches the web UI for a long time, thus no
failed items will show up for a long time. Now errors are populated when the
web UI is opened.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4650
LGTM: AudriusButkevicius
2018-01-14 17:01:06 +00:00
Lars K.W. Gohlke
89a021609b lib/scanner: Refactoring
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4642
LGTM: imsodin, AudriusButkevicius
2018-01-14 14:30:11 +00:00
Emil Hessman
bbc178ccc4 lib/fs: harmonize CreateSymlink definitions (fixes #4567)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4632
LGTM: imsodin, AudriusButkevicius
2018-01-14 14:25:04 +00:00
Simon Frei
341b9691a7 lib/connections, lib/model: Additional connection info in logs (fixes #4499)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4665
2018-01-12 11:27:55 +00:00
Jakob Borg
6e0f64017a lib/logger, lib/model: Correct printf format strings in tests
This is a build (test) failure in Go 1.10.
2018-01-12 08:27:00 +01:00
Jakob Borg
14df5211af lib/model: Correctly account handling of sparse blocks (fixes #4657)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4658
2018-01-11 10:36:35 +00:00
Antony Male
db03562d43 lib/scanner: Fix UTF-8 normalization on ZFS (fixes #4649)
It turns out that ZFS doesn't do any normalization when storing files,
but does do normalization "as part of any comparison process".

In practice, this seems to mean that if you LStat a normalized filename,
ZFS will return the FileInfo for the un-normalized version of that
filename.

This meant that our test to see whether a separate file with a
normalized version of the filename already exists was failing, as we
were detecting the same file.

The fix is to use os.SameFile, to see whether we're getting the same
FileInfo from the normalized and un-normalized versions of the same
filename.

One complication is that ZFS also seems to apply its magic to os.Rename,
meaning that we can't use it to rename an un-normalized file to its
normalized filename. Instead we have to move via a temporary object. If
the move to the temporary object fails, that's OK, we can skip it and
move on. If the move from the temporary object fails however, I'm not
sure of the best approach: the current one is to leave the temporary
file name as-is, and get Syncthing to syncronize it, so at least we
don't lose the file. I'm not sure if there are any implications of this
however.

As part of reworking normalizePath, I spotted that it appeared to be
returning the wrong thing: the doc and the surrounding code expecting it
to return the normalized filename, but it was returning the
un-normalized one. I fixed this, but it seems suspicious that, if the
previous behaviour was incorrect, noone ever ran afoul of it. Maybe all
filesystems will do some searching and give you a normalized filename if
you request an unnormalized one.

As part of this, I found that TestNormalization was broken: it was
passing, when in fact one of the files it should have verified was
present was missing. Maybe this was related to the above issue with
normalizePath's return value, I'm not sure. Fixed en route.

Kindly tested by @khinsen on the forum, and it appears to work.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4646
2018-01-05 18:11:09 +00:00
Jakob Borg
992bb0a98c lib/config, lib/discover: Support new discovery cluster (ref #4618)
This adds one new feature, that discovery servers can have ?nolookup to
be used only for announces. The default set of discovery servers is
changed to:

- discovery.s.n used for lookups. This is dual stack load balanced over
  all discovery servers, and returns both IPv4 and IPV6 results when they
  exist.

- discovery-v4.s.n used for announces. This has IPv4 addresses only and
  the discovery servers will update the unspecified address with the IPv4
  source address, as usual.

- discovery-v6.s.n which is exactly the same for IPv6.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4647
2018-01-05 14:18:32 +00:00
Audrius Butkevicius
53f4dfe83c lib/model: Fix panic when auto-accepting existing paused folder (fixes #4636)
This no longer pokes at model internals, and only touches the config.
As a result, model handles this in CommitConfiguration, which restarts
the folders if things change, which repopulate m.folderDevice, m.deviceFolder
and other interal mappings.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4639
2018-01-03 07:42:25 +00:00
Audrius Butkevicius
b0e2050cdb cmd/syncthing: UI for version restoration (fixes #2599) (#4602)
cmd/syncthing: Add UI for version restoration (fixes #2599)
2018-01-01 15:39:23 +01:00
Audrius Butkevicius
c7f136c2b8 lib/upnp: Each service is it's own NAT device
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4625
2017-12-30 19:16:08 +00:00
Jakob Borg
a9f0659f2f lib/fs: Handle deduplicated files on NTFS (fixes #1845)
These files always have the symlink bit set, because they are reparse
points. Nonetheless they are not symlinks, and Lstat reports a size for
them. We use this fact to disambiguate, and hope fervently that nothing
else matches this description so it comes back to bite us...

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4622
2017-12-29 21:23:06 +00:00
Lars K.W. Gohlke
9988044bbe lib/model: Cleanup conditions
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4621
2017-12-29 13:14:39 +00:00
Audrius Butkevicius
72172d853c vendor: Move back to upstream KCP (fixes #4407)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4614
2017-12-27 11:33:12 +00:00
Simon Frei
841205dbfe lib/model: Check for invalid filenames after ignore patterns
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4607
LGTM: calmh, AudriusButkevicius
2017-12-25 17:54:34 +00:00
Audrius Butkevicius
c58b383b6d gui: Add debug tab to settings (ref #2644)
Just because there are a ton of people struggling to set env vars.
Perhaps this should live in advanced settings, and perhaps we should have a button to view the log.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4604
LGTM: calmh, imsodin
2017-12-24 22:26:05 +00:00
Audrius Butkevicius
2547a29dd7 lib/connections: Don't close nil connections (fixes #4605) 2017-12-18 14:40:51 +00:00
Simon Frei
8fa2b7765a gui, lib/model: Display list of files needed by remote (fixes #4369)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4559
LGTM: AudriusButkevicius, calmh
2017-12-15 20:01:56 +00:00
Jakob Borg
c7522063b3 lib/model: Don't leak fd when truncate fails (fixes #4593)
Also attempt to handle this nicer by ignoring the truncate failure when
it doesn't matter, and recover by deleting the temp file when it does.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4594
2017-12-14 10:42:40 +00:00
Jakob Borg
d1d967f0cf lib/db: Keep folder meta data persistently in db (fixes #4400)
This keeps the data we need about sequence numbers and object counts
persistently in the database. The sizeTracker is expanded into a
metadataTracker than handled multiple folders, and the Counts struct is
made protobuf serializable. It gains a Sequence field to assist in
tracking that as well, and a collection of Counts become a CountsSet
(for serialization purposes).

The initial database scan is also a consistency check of the global
entries. This shouldn't strictly be necessary. Nonetheless I added a
created timestamp to the metadata and set a variable to compare against
that. When the time since the metadata creation is old enough, we drop
the metadata and rebuild from scratch like we used to, while also
consistency checking.

A new environment variable STCHECKDBEVERY can override this interval,
and for example be set to zero to force the check immediately.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4547
LGTM: imsodin
2017-12-14 09:51:17 +00:00
Jakob Borg
8c91ced784 cmd/syncthing: Clean up deadlock envvars
So STDEADLOCK seems to do the same thing as STDEADLOCKTIMEOUT, except in
the other package. Consolidate?

STDEADLOCKTHRESHOLD is actually called STLOCKTHRESHOLD, correct the help
text.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4598
2017-12-13 19:40:12 +00:00
Jakob Borg
230fb083fc lib/model: Remove dead code (*sharedPullerState).sourceFile 2017-12-12 11:30:47 +01:00
Pawel Palenica
ece1defb2f lib/dialer: Register dialer for socks URL scheme (fixes #4515)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4579
2017-12-08 12:04:43 +00:00
Simon Frei
cce634f340 lib/model: Improve scan scheduling and dir del during pull (fixes #4475 #4476)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4493
2017-12-07 08:42:03 +00:00
Jakob Borg
47429d01e8 lib/config, lib/model: Tweaks to the auto accept feature
Fix the folder restart behavior (ignore Label), improve the API for that
(imho).

Also removes the tab switch animation in the settings modal, because
annoying.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4577
2017-12-07 08:33:32 +00:00
Audrius Butkevicius
445c4edeca gui, lib/config, lib/model: Support auto-accepting folders (fixes #2299)
Also introduces a new Waiter interface for config changes and segments the
configuration GUI.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4551
2017-12-07 07:08:24 +00:00
Simon Frei
c005b8dcb0 lib/fs: Prolong test timeout on darwin, hopefully fixing flakyness
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4576
2017-12-06 22:07:08 +00:00
Audrius Butkevicius
95a65bf0d0 lib/config: Support symlinked root (fixes #4542, fixes #4353)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4545
LGTM: imsodin, calmh
2017-11-26 07:51:22 +00:00
Jakob Borg
429b3a0429 lib/osutil, lib/scanner: Run symlink test on Windows when possible
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4548
2017-11-25 21:49:53 +00:00
Thomas Hipp
b2af8f135b lib/events: Fix unmarshaling of EventType
skip-check: authors

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4540
2017-11-22 23:25:55 +00:00
Simon Frei
ce29d3a574 all: Various debug logging improvements
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4529
2017-11-22 08:05:27 +00:00
Jakob Borg
6daa766fde lib/connections: Actually fix LAN detection, for real (ref #4534) 2017-11-22 09:01:21 +01:00
Audrius Butkevicius
4922b46fbd lib/connections: Fix local address priority
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4534
LGTM: imsodin, calmh
2017-11-22 07:05:49 +00:00
xjtdy888
a17d953334 lib/connections: Actually make connection attempts for lower priority addresses as well
Skip-check: authors

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4535
2017-11-21 14:58:18 +00:00
Audrius Butkevicius
44a542391e lib/connections: Trust the model to tell us if we are connected
This should address issue as described in https://forum.syncthing.net/t/stun-nig-party-with-paused-devices/10942/13
Essentially the model and the connection service goes out of sync in terms of thinking if we are connected or not.
Resort to model as being the ultimate source of truth.

I can't immediately pin down how this happens, yet some ideas.

ConfigSaved happens in separate routine, so it's possbile that we have some sort of device removed yet connection comes in parallel kind of thing.
However, in this case the connection exists in the model, and does not exist in the connection service and the only way for the connection to be removed
in the connection service is device removal from the config.

Given the subject, this might also be related to the device being paused.

Also, adds more info to the logs

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4533
2017-11-21 07:25:38 +00:00
Jakob Borg
0901350087 lib/model: Trigger a pull when ignore patterns change
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4532
2017-11-20 16:29:36 +00:00
Jakob Borg
72d645865e lib/connections: Fix race condition in parallel dial, minor cleanups (fixes #4526)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4527
2017-11-19 17:38:13 +00:00
Dmitry Saveliev
9471b9f6af lib/versioner: Purge the empty directories in .stversions (fixes #4406)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4514
LGTM: AudriusButkevicius, imsodin
2017-11-18 15:56:53 +00:00
Audrius Butkevicius
0518a92cdb lib/connections: Only announce punchable nats (fixes #4519)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4523
2017-11-17 14:46:45 +00:00
Simon Frei
6cf01c1d30 lib/model: Don't update ignore hash when pull fails
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4522
2017-11-17 12:42:41 +00:00
Jakob Borg
5f4ed66aa1 lib/model: Properly schedule pull on reconnect (fixes #4504)
We need to reset prevSeq so that we force a full check when someone
reconnects - the sequence number may not have changed due to the
reconnect. (This is a regression; we did this before f6ea2a7.)

Also add an optimization: we schedule a pull after scanning, but there
is no need to do so if no changes were detected. This matters now
because the scheduled pull actually traverses the database which is
expensive.

This, however, makes the pull not happen on initial scan if there were
no changes during the initial scan. Compensate by always scheduling a
pull after initial scan in the rwfolder itself.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4508
LGTM: imsodin, AudriusButkevicius
2017-11-17 12:11:45 +00:00
Jakob Borg
ee5d0dd43f lib/fs: Add case insensitivity to MtimeFS
This is step one of a hundred fifty on the path to case insensitivity.
It brings in the basic case folding mechanism and adds it to the
mtimefs, as this is something outside the fileset that touches stuff in
the database based on name. No effort to convert or handle existing
entries when the insensitivity is changed, I don't think we need it...

Useless by itself but includes tests and will reduce the review load
along the way.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4521
2017-11-17 12:10:16 +00:00
Jakob Borg
7ebf58f1bc Fix discovery in the absence of listen addresses (fixes #4418)
This makes it OK to not have any listeners working. Specifically,

- We don't complain about an empty listener address
- We don't complain about not having anything to announce to global
  discovery servers
- We don't send local discovery packets when there is nothing to
  announce.

The last point also fixes a thing where the list of addresses for local
discovery was set at startup time and never refreshed.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4517
2017-11-17 09:12:35 +00:00
AudriusButkevicius
aecd7c64ce lib/connections: Parallel dials in the same priority (fixes #4456)
Well Tested(TM)

Introduces a potential issue where we always pick some connectable but dodgy connection that breaks
soon after the TLS handshake.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4489
2017-11-15 09:36:33 +00:00
Simon Frei
4efff736b3 lib/connections: Consistent log levels & polish (fixes #4510)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4511
2017-11-14 21:49:36 +00:00
Simon Frei
fa12a18190 lib/model: Handle type changes when pulling (ref #4505 #4506 #4507)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4509
LGTM: AudriusButkevicius, calmh
2017-11-13 15:16:27 +00:00
Jakob Borg
2b65e1062e lib/model: Fix rescan detection (fixes #4505, fixes #4506)
Diff is large due to comment reformatting and indentation but all it
does is wrap the file mtime/size/permissions check in an "if
stat.IsRegular()".

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4507
2017-11-13 06:57:07 +00:00
Jakob Borg
6e148e20cd test: Mend tests for latest event changes etc 2017-11-13 00:25:07 +01:00
Jakob Borg
d7d45d8092 lib/db: Refactor away the large genericReplace thing
This removes a significant, complex chunk of database code. The
"replace" operation walked both the old and new in lockstep and made the
relevant changes to make the new situation correct. But since delta
indexes we pretty much never need this - we just used replace to drop
the existing data and start over.

This makes that explicit and removes the complexity.

(This is one of those things that would be annoying to make case
insensitive, while the actual "drop and then insert" that we do is
easier.)

This is fairly well unit tested...

The one change to the tests is to cover the fact that previously replace
with something identical didn't bump the sequence number, while
obviously removing everything and re-inserting does. This is not
behavior we depend on anywhere.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4500
LGTM: imsodin, AudriusButkevicius
2017-11-12 20:20:34 +00:00
Simon Frei
c080f677cb all: Add invalid/ignored files to global list, announce to peers (fixes #623)
This lets us determine accurate completion status for remote peers when they
have ignored files.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4460
2017-11-11 19:18:17 +00:00
Simon Frei
ec4c3bae0d lib/watchaggregator: Don't care about timings during testing on darwin
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4502
2017-11-10 17:05:31 +00:00
Audrius Butkevicius
386cb274bd lib/model: Incremental block stats usage reporting
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4498
LGTM: calmh
2017-11-09 21:16:29 +00:00
Jakob Borg
a9d422e008 Revert "lib/model: Use the new file name when requesting rescan"
This has consequences that I haven't thought through for case mismatches
(i.e. we have foo.txt in the database and queue a rescan for FOO.txt).
2017-11-09 13:55:45 +01:00
Jakob Borg
75f64733f9 lib/model: Use the new file name when requesting rescan
If the file is not already in the database, the "curfile" will be empty
and the rescan request cover the entire folder.
2017-11-09 13:48:39 +01:00
AudriusButkevicius
0ee1146e1c lib/connections: Indicate stack in transport (fixes #4463)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4486
2017-11-07 07:25:05 +00:00
Simon Frei
f6ea2a7f8e lib/model: Trigger pulls instead of pulling periodically
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4340
2017-11-07 06:59:35 +00:00
AudriusButkevicius
62a4106a79 lib/connections: Fix lan detection (fixes #4421)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4487
2017-11-06 14:05:29 +00:00
Jakob Borg
9c855ab22e lib/config, lib/model: Configurable folder marker name (fixes #1126)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4483
2017-11-05 12:18:05 +00:00
HairyFotr
7cbd92e1b1 all: Fix comment typos
Skip-check: authors

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4481
2017-11-04 07:20:11 +00:00
Simon Frei
b999b58049 lib/watchaggregator: Relax test timing requirement for the benefit of macos
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4469
2017-10-26 21:09:10 +00:00
Simon Frei
4b69d0e093 lib/model: Add initial noop watch cancel func (fixes #4464)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4465
2017-10-26 11:49:06 +00:00
Simon Frei
7ba9e7c322 lib/db: Filter unchanged files when updating and polish
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4426
2017-10-24 20:05:29 +00:00
Simon Frei
dc42db444b lib/model, lib/config: Refactor folder health/error handling (fixes #4445, fixes #4451)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4455
LGTM: AudriusButkevicius, calmh
2017-10-24 07:58:55 +00:00
Audrius Butkevicius
a9c221189b lib/connections: Stun resolves server adress beforehand (fixes #4453)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4454
2017-10-22 18:48:06 +00:00
Audrius Butkevicius
b2966957e0 lib/connections: Add KCP blacklist period 2017-10-22 13:56:52 +01:00
Audrius Butkevicius
0d30166357 lib/connections: Use own KCP fork, move listener setup earlier (ref #4446)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4452
2017-10-22 12:36:36 +00:00
Jakob Borg
d65f1fb08a lib/config: Improve debug logging around folder marker 2017-10-22 00:04:51 +02:00
Audrius Butkevicius
622b614f31 all: Ignore Sync errors on directories (fixes #4432)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4449
2017-10-21 22:00:46 +00:00
Simon Frei
e70003737b lib/fs: make watcher tests even more darwin slowness resistant
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4439
2017-10-20 15:59:18 +00:00
Michael Ploujnikov
f98c21b68e all: Add filesystem notification support
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3986
2017-10-20 14:52:55 +00:00
Audrius Butkevicius
fb7264a663 cmd/syncthing: Enable KCP by default
Also, use upstream library, as my changes have been merged.
2017-10-17 23:17:10 +01:00
Audrius Butkevicius
becbb3b123 lib/model: Fix tests not to require Go 1.9 2017-10-12 22:36:20 +01:00
Jakob Borg
2f08f8021f lib/beacon: Don't exit after a single write failure (fixes #4414)
With VPNs and stuff we can get a single failure on an interface that
supposedly supports broadcasts without it being fatal.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4415
2017-10-12 07:13:44 +00:00
Audrius Butkevicius
9d3f3847ed lib/model: Fix removal of paused folders, improve tests (fixes #4405) 2017-10-12 08:23:33 +02:00
Audrius Butkevicius
74c8d34805 lib/model: Centralize error reporting, modified files are errors (fixes #4392) 2017-10-12 08:23:33 +02:00
Audrius Butkevicius
5ec1490be0 lib/fs: Ignore directory fsync failures 2017-10-12 08:22:29 +02:00
Audrius Butkevicius
2760d032ca cmd/syncthing: Add more stats to usage reports (ref #3628)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4347
2017-10-12 06:16:46 +00:00
Jakob Borg
a365ae51c4 lib/model: Hide temporary files on Windows while they are in use (fixes #4382)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4394
2017-09-23 13:47:51 +00:00
Jakob Borg
97222797a0 lib/config: Make folder marker change non fatal 2017-09-23 15:29:55 +02:00
Simon Frei
e588bb29b9 lib/model: remove unused folderFs member from Model
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4389
2017-09-21 09:03:36 +00:00
Jakob Borg
2dd9450793 lib/protocol, vendor: Import luhn code directly
I've changed it incompatibly to fix a correctness bug. Nonetheless, we
should remain incorrect indefinitely.
2017-09-20 21:34:32 +02:00
Jakob Borg
3ee12464b4 lib/config, lib/model: Make sure to hide our special files (fixes #4382)
The folder marker conversion forgot to hide the .stfolder. This adds
that, for those who have not yet been converted.

Also adds Hide() calls to the folder start, to mend historical
unhidedness. (I'm sure this will upset someone who is manually managing
their .stignores in the other direction...)

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4384
2017-09-20 06:49:04 +00:00
Simon Frei
1088eb12ea lib/model: Fix logging inconsistencies (fixes #4375)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4376
2017-09-18 11:56:19 +00:00
Jakob Borg
429cc20eb7 cmd/syncthing: Add some common security releated HTTP headers (fixes #4360)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4361
2017-09-10 08:28:12 +00:00
Audrius Butkevicius
e85ce7c94e lib/model: Support removing paused folders (fixes #4357)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4358
LGTM: imsodin, calmh
2017-09-09 15:08:59 +00:00
Jakob Borg
283c8d95e2 lib/protocol: Comment typo 2017-09-08 15:25:14 +02:00
wangguoliang
a9aa375109 lib/fs: Comment typo
Skip-check: authors

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4354
2017-09-07 09:40:46 +00:00
Simon Frei
f7d2c58783 lib/model: Deduplicate folder loops
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4352
LGTM: AudriusButkevicius, calmh
2017-09-07 06:17:47 +00:00
Jakob Borg
4d3e0de4ba cmd/syncthing, lib/sha256: Skip CPU benchmarks when user decided (fixes #4348)
When STHASHING is set, don't benchmark as it's already decided. If weak
hashing isn't set to "auto", don't benchmark that either.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4349
2017-09-06 06:55:47 +00:00
Simon Frei
9dbc509996 lib/ignore: Consistent behaviour for nil *Matcher
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4310
2017-09-06 06:39:18 +00:00
Simon Frei
c41aaad3bb lib/ignore: Ignore duplicate lines in .stignore
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4350
LGTM: AudriusButkevicius, calmh
2017-09-04 12:46:19 +00:00
Jakob Borg
9682bbfbda lib/protocol: Optimize luhn and chunk functions
These functions were very naive and slow. We haven't done much about
them because they pretty much don't matter at all for Syncthing
performance. They are however called very often in the discovery server
and these optimizations have a huge effect on the CPU load on the
public discovery servers.

The code isn't exactly obvious, but we have good test coverage on all
these functions.

benchmark                 old ns/op     new ns/op     delta
BenchmarkLuhnify-8        12458         1045          -91.61%
BenchmarkUnluhnify-8      12598         1074          -91.47%
BenchmarkChunkify-8       10792         104           -99.04%

benchmark                 old allocs     new allocs     delta
BenchmarkLuhnify-8        18             1              -94.44%
BenchmarkUnluhnify-8      18             1              -94.44%
BenchmarkChunkify-8       44             2              -95.45%

benchmark                 old bytes     new bytes     delta
BenchmarkLuhnify-8        1278          64            -94.99%
BenchmarkUnluhnify-8      1278          64            -94.99%
BenchmarkChunkify-8       42552         128           -99.70%

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4346
2017-09-03 10:26:12 +00:00
Audrius Butkevicius
9e6a1fdcd4 vendor: Update kcp, removes closeConn (fixes #4343) 2017-09-02 16:11:48 +02:00
Simon Frei
55e0ac3e24 lib/model: Make puller retrying and logging less aggressive
Currently all errors during pulling and the first of these errors again on
finishing are logged to info. Besides that the errors logged when finishing
are stored in f.errors. This PR moves all logging during pulling to the debug
channel (they might still be relevant in some obscure debugging case) and
uses the stored errors to log the main error per fail when all pulling
iterations are done and failed.

Additional instead of trying 11 times it now only tries 3 times.

This is the first part of what is discussed here:
https://forum.syncthing.net/t/reduce-verboseness-of-puller/10261

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4338
2017-09-02 06:05:55 +00:00
Audrius Butkevicius
cbcc3ea132 lib/connections: Use our own fork of kcp (fixes #4063)
This updates kcp and uses our own fork which:

1. Keys sessions not just by remote address, but by remote address +
conversation id 2. Allows not to close connections that were passed directly
to the library. 3. Resets cache key if the session gets terminated.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4339
LGTM: calmh
2017-09-02 06:04:35 +00:00
Audrius Butkevicius
ab132ff6fe lib: Folder marker is now a folder
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4341
LGTM: calmh
2017-09-02 05:52:38 +00:00
Jakob Borg
a8145e187f lib/model: Fixup because I suck and shouldn't commit directly to master 2017-08-31 10:49:17 +02:00
Jakob Borg
e33fa10115 lib/model: Use same batch size constants in db updater as for protocol 2017-08-31 10:47:39 +02:00
Jakob Borg
4b6e7e7867 lib/tlsutil: Remove undesired bufio from UnionedConnection (ref #4245)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4335
2017-08-31 07:34:48 +00:00
Simon Frei
ddf6d64faa lib/model: Create folders via newFolder
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4329
2017-08-25 19:47:01 +00:00
Audrius Butkevicius
a69ba18f62 lib/model: Some platforms do not support usage checks (fixes #4321)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4322
2017-08-22 18:13:58 +00:00
Simon Frei
0ca0e3e9bd lib/model: GetIgnores: Don't return error for no .stignore file
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4311
2017-08-22 06:48:25 +00:00
Audrius Butkevicius
0a96a1150b lib/model, lib/ignores: Properly handle out of folder ignores and free space checks (fixes #4313, fixes #4314)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4318
2017-08-22 06:45:00 +00:00
Audrius Butkevicius
b8c249cddc lib/model: Move stale scan check info finisher (ref #4305, fix #3742)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4317
2017-08-22 06:42:09 +00:00
Audrius Butkevicius
e8ba6d4771 lib/upnp: Fix build
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4316
2017-08-21 11:41:40 +00:00
Audrius Butkevicius
606fce09ca lib/upnp: Disable confusing messages
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4312
2017-08-21 10:03:25 +00:00
Audrius Butkevicius
3d8b4a42b7 all: Convert folders to use filesystem abstraction
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4228
2017-08-19 14:36:56 +00:00
Simon Frei
ab8c2fb5c7 lib/model: Fix race in GetIgnores (fixes #4300)
In addition this function returned an error when .stignore file was not
present, which is perfectly valid. Also removed inconsistent nil check in
ignores.go (only relevant for tests) and adjusted walk.go to do what it says
in comments (check if Matcher is nil) to prevent nil deref in tests.

Originally reported in:
https://forum.syncthing.net/t/reason-for-panic-maybe-too-little-ram/10346

Regression from #3996

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4301
2017-08-12 17:10:43 +00:00
Jakob Borg
1fc2ab444b lib/model: Remove ineffective symlink recovery attempt 2017-08-08 15:30:28 +02:00
Jakob Borg
fa5c890ff6 lib/versioner: Clean the versions dir of symlinks, not the full folder
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4289
2017-08-08 13:13:08 +00:00
Jakob Borg
a3c17f8f81 lib/model: Disable symlink attack test on Windows 2017-08-08 08:05:24 +02:00
Jakob Borg
f1f21bf220 lib/model, lib/versioner: Prevent symlink attack via versioning (fixes #4286)
Prior to this, the following is possible:

- Create a symlink "foo -> /somewhere", it gets synced
- Delete "foo", it gets versioned
- Create "foo/bar", it gets synced
- Delete "foo/bar", it gets versioned in "/somewhere/bar"

With this change, versioners should never version symlinks.
2017-08-07 07:57:10 +02:00
Simon Frei
a04b92332f gui, lib/config: Add default path for new folders (fixes #2157)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4192
2017-07-20 13:16:54 +00:00
HairyFotr
0ad10b0fee all: Typos
Skip-check: authors

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4263
2017-07-20 13:10:46 +00:00
Simon Frei
b75c9f2bbb lib/ignores: Don't add text from includes to lines (fixes #4249)
Otherwise all the lines from includes will be shown in the web UI instead of
just the #include ... line. This problem was introduced in #3996.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4248
LGTM: calmh
2017-07-06 11:44:11 +00:00
Siyuan Liu
03c678a810 lib/versioner: Interpret versions path relative to folder path (fixes #4188)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4243
2017-07-03 14:50:51 +00:00
Jakob Borg
add10c98fa lib/db: Fix test for Go 1.9's smarter time.Time values 2017-06-28 14:36:04 +02:00
Jakob Borg
db1dc9985a lib/upgrade: 0.x to 1.0 is a minor upgrade
This removes the special handling of minor versions as major when the
actual major is zero, and adds the special case that upgrades from 0.x
to 1.x are considered minor. 0.x to 2.x or 1.x to 2.x etc are still
considered major.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4226
2017-06-25 14:17:43 +00:00