Commit Graph

130 Commits

Author SHA1 Message Date
Jakob Borg
aedc2d788f
lib/model, lib/versioner: Drive version cleanup from scanner (fixes #6313) (#6693)
This change adds a separate config for the cleanup interval, and runs that cleanup from the main folder loop.
2020-07-14 10:48:50 +02:00
Simon Frei
0648fb0626
lib/model: Don't force rescan already changed items (#6798) 2020-06-25 20:23:59 +02:00
Simon Frei
f66e57947b
lib/model: Use right db snap when scanning recvonly folder (#6769) 2020-06-21 09:28:29 +02:00
Simon Frei
22f0077262
lib/model: Don't stay scanning forever on fail (#6761) 2020-06-18 14:13:46 +01:00
Simon Frei
8cf9d91ed4
lib: Print nicely rounded durations (#6756) 2020-06-18 10:55:41 +02:00
Simon Frei
a47546a1f1
lib/scanner, lib/model: Improve error handling when scanning (#6736) 2020-06-16 09:25:41 +02:00
Simon Frei
1f8e6c55f6
lib/db: Refactor to use global list by version (fixes #6372) (#6638)
Group the global list of files by version, instead of having one flat list for all devices. This removes lots of duplicate protocol.Vectors.

Co-authored-by: Jakob Borg <jakob@kastelo.net>
2020-05-30 09:50:23 +02:00
Simon Frei
ed6bfc5417
lib/model: Don't increase pull pause on triggered pull (#6690) 2020-05-29 09:52:28 +02:00
Jakob Borg
651ee2ce74
lib/model: Partial revert of rename fix (fixes #6653) (#6656)
We can't just drop the snap because it's in use elsewhere. This should
be equally functional.
2020-05-16 23:05:33 +02:00
Audrius Butkevicius
a1c5b44c74
lib/model: Fix rename handling (ref #6650) (#6652) 2020-05-16 14:39:27 +02:00
Simon Frei
974551375e
lib/db: Dont add symlinks to blocks map (fixes #6637) (#6639) 2020-05-13 20:38:21 +02:00
Audrius Butkevicius
decb967969
all: Reorder sequences for better rename detection (#6574) 2020-05-11 20:15:11 +02:00
Jakob Borg
744ef0d8ac
lib/protocol: Avoid data loss on database wipe by higher version numbers (fixes #3876) (#6605)
This makes version vector values clock based instead of just incremented
from zero. The effect is that a vector that is created from scratch
(after database reset) will have a higher value for the local device
than what it could have been previously, causing a conflict. That is, if
we are A and we had

    {A: 42, B: 12}

in the old scheme, a reset and rescan would give us

    {A: 1}

which is a strict ancestor of the older file (this might be wrong). With
the new scheme we would instead have

    {A: someClockTime, b: otherClockTime}

and the new version after reset would become

    {A: someClockTime+delta}

which is in conflict with the previous entry (better).
In case the clocks are wrong (current time is less than the value in the
vector) we fall back to just simple increment like today.

This scheme is ineffective if we suffer a database reset while at the
same time setting the clock back far into the past. It's however no
worse than what we already do.

This loses the ability to emit the "added" event, as we can't look for
the magic 1 entry any more. That event was however already broken
(#5541).

Another place where we infer meaning from the vector itself is in
receive only folders, but there the only criteria is that the vector is
one item long and includes just ourselves, which remains the case with
this change.

* wip
2020-05-06 08:47:02 +02:00
Simon Frei
914eb77ca4
lib/model: Don't include iolimiter wait into sync duration (#6593) 2020-05-04 08:43:35 +02:00
Audrius Butkevicius
bd0c2bf237
lib/model: Do file recheck in folder loop (fixes #6583) (#6585) 2020-05-01 11:08:59 +02:00
Simon Frei
6bbd24de12
lib/model: Refactor folder health/error handling (fixes #6557) (#6558) 2020-04-21 10:15:59 +02:00
Simon Frei
49798552f2
lib/model: Delay watch setup on errors (fixes #5731) (#6544) 2020-04-17 17:43:42 +02:00
Simon Frei
32245435e2
lib/model: Handle deleted items on RO for remote removes (fixes #6432) (#6464) 2020-04-02 16:14:25 +02:00
Simon Frei
79a758be3c
lib/model: Do Revert/Override synchronously (#6460) 2020-03-27 13:05:09 +01:00
Simon Frei
f0e33d052a
lib: More contextification (#6343) 2020-02-24 21:57:15 +01:00
Simon Frei
fae7425bbf
lib: Modify FileInfos consistently (ref #6321) (#6349) 2020-02-19 16:58:09 +01:00
Jakob Borg
b61da487e4 all: Update metadata modtime on delete (ref #6284) (#6315)
This records the time a file was marked as deleted. It will, in the
future, aid in garbage collecting old files.
2020-02-10 10:48:30 +01:00
Jakob Borg
55937b61ca
lib/model: Add global request limiter (fixes #6302) (#6303)
This adds a new config with the simple and concise name
maxConcurrentIncomingRequestKiB. This limits how many bytes we have "in
the air" in the form of response data being read and processed.

After some testing I think that not having this limiter is seldom a
great idea and thus I propose a default value of 256 MiB for this new
setting.

I also refactored the folder IO limiter to be a model/folder attribute
instead of a package global.
2020-02-01 08:02:18 +01:00
Jakob Borg
d91c4b010b
lib/config, lib/model: Limit concurrent pulls (fixes #5914) (#6290)
Adds a new folder state "Waiting to Sync" in the same vein as the
existing "Waiting to Scan". This vastly improves performances in the
rare cases when there are lots and lots of folders operating.
2020-01-27 17:31:17 +01:00
Simon Frei
08f0e125ef all: Transactionalize db.FileSet (fixes #5952) (#6239) 2020-01-21 18:23:08 +01:00
Simon Frei
90d85fd0a2
lib: Replace done channel with contexts in and add names to util services (#6166) 2019-11-21 08:41:15 +01:00
Simon Frei
0d14ee4142
lib/model: Don't info log repeat pull errors (#6149) 2019-11-19 09:56:53 +01:00
Simon Frei
a2a14c8424 lib/model: Set empty version when unignoring deleted files (fixes 6038) (#6039) 2019-10-01 15:34:59 +02:00
Simon Frei
28b6e8b063
lib/db: Update db when only local flags change (fixes #6008) (#6007) 2019-09-12 08:47:39 +02:00
Jakob Borg
ebd2e5bf30
lib/model: Correctly handle manual rescan with ignore error (fixes #5985) (#5987)
Assume a folder error was set due to bad ignores on the latest scan.
Previously, doing a manual rescan would result in:

1. Clearing the folder error, which schedules (immediately) an fs
   watcher restart

2. Attempting to load the ignores, which fails, so we set a folder
   error and bail.

3. Now the fs watcher restarts, as scheduled, so we trigger a scan.
   Goto 1.

This change fixes this by not clearing the error until the error is
actually cleared, that is, if both the health check and ignore loading
succeeds.
2019-08-30 13:27:26 +01:00
Jakob Borg
24d4290d03 lib/model, lib/scanner: Pass a valid event logger (fixes #5970) (#5971) 2019-08-21 08:05:43 +02:00
Simon Frei
b1c74860e8
all: Remove global events.Default (ref #4085) (#5886) 2019-08-15 16:29:37 +02:00
Simon Frei
6fa02d5081 lib/model: Fix a few more problematic locks (ref #5929) (#5944) 2019-08-13 09:04:43 +02:00
Simon Frei
7a4c88d4e4 lib: Add mtime window when comparing files (#5852) 2019-07-23 21:48:53 +02:00
Simon Frei
942659fb06
lib/model, lib/nat: More service termination speedup (#5884) 2019-07-23 10:49:22 +02:00
Simon Frei
3446d50201 lib/model: Remove pointless error that watch hasn't started (fixes #5833) (#5834) 2019-07-10 11:00:06 +02:00
Simon Frei
ba056578ec
lib: Add util.Service as suture.Service template (fixes #5801) (#5806) 2019-07-09 11:40:30 +02:00
Simon Frei
3c7e7e971d lib/model: Make jobQueue.Jobs paginated (fixes #5754) (#5804)
* lib/model: Make jobQueue.Jobs paginated (fixes #5754)

* fix, no test yet

* add test
2019-06-27 19:25:38 +01:00
Jakob Borg
997bb5e7e1 all: Remove "large blocks" config (#5763)
We now always use large / variable blocks.
2019-06-06 15:57:38 +01:00
Simon Frei
486230768e lib/fs, lib/model: Add error channel to Watch to avoid panics (fixes #5697) (#5734)
* lib/fs, lib/model: Add error channel to Watch to avoid panics (fixes #5697)

* forgot unsupported watch

* and more non(-standard)-unixy fixes

* and windows test

* review
2019-05-25 20:08:26 +01:00
Simon Frei
9b2a73f9ab
lib/model: Pause pull for at least as long as failed pull took (fixes #5641) (#5643) 2019-04-10 16:22:23 +02:00
Simon Frei
395e524e2d lib/model: Update db on scan/pull in folder (#5608) 2019-04-07 13:29:17 +02:00
Simon Frei
289a02e994 lib/model: Integrate stat refs in folder (#5596) 2019-03-11 16:57:21 +00:00
Simon Frei
445637ebec lib/model: Pass fset & ignores on folder creation (#5592) 2019-03-11 07:28:54 +01:00
Simon Frei
722b3fce6a all: Hide implementations behind interfaces for mocked testing (#5548)
* lib/model: Hide implementations behind interfaces for mocked testing

* review
2019-02-26 08:09:25 +00:00
Simon Frei
4299af1c63 lib/config, lib/model: Use path from locations to check disk space for db (#5525) 2019-02-12 12:25:11 +00:00
Simon Frei
47e08797cb lib/model: Don't pull if ignores failed to load and cleanup (#5418) 2019-01-01 10:17:14 +01:00
Simon Frei
8bb9878f26 lib/model: Check folder context before setting error state (#5416) 2018-12-30 21:56:16 +01:00
Simon Frei
fc81e2b3d7 lib/model: Do folder watch operations under lock (fixes #5392) (#5395) 2018-12-21 12:06:21 +01:00
Audrius Butkevicius
ff2cde469e lib/model: Allow limiting number of concurrent scans (fixes #2760) (#4888) 2018-12-05 08:40:05 +01:00
Simon Frei
2f9840ddae lib: Introduce fs.IsParent (fixes #5324) (#5326) 2018-11-22 11:16:45 +01:00
Simon Frei
d510e3cca3 all: Display errors while scanning in web UI (fixes #4480) (#5215) 2018-11-07 11:04:41 +01:00
Jakob Borg
f528923d1e lib/model, cmd/syncthing: Wait for folder restarts to complete (fixes #5233) (#5239)
* lib/model, cmd/syncthing: Wait for folder restarts to complete (fixes #5233)

This is the somewhat ugly - but on the other hand clear - fix for what
is really a somewhat thorny issue. To avoid zombie folder runners a new
mutex is introduced that protects the RestartFolder operation. I hate
adding more mutexes but the alternatives I can think of are worse.

The other part of it is that the POST /rest/system/config operation now
waits for the config commit to complete. The point of this is that until
the commit has completed we should not accept another config commit. If
we did, we could end up with two separate RestartFolders queued in the
background. While they are both correct, and will run without
interfering with each other, we can't guarantee the order in which they
will run. Thus it could happen that the newer config got committed
first, and the older config commited after that, leaving us with the
wrong config running.

* test

* wip

* hax

* hax

* unflake test

* per folder mutexes

* paranoia

* race
2018-10-05 09:26:25 +01:00
Simon Frei
cb0f4ce55a lib/model: Don't stop folder if out of disk space (fixes #2370) (#5099)
This removes the out of disk space check from CheckHealth. The disk space is now
only checked if there are files to pull, in which case pulling those files is
stopped, but everything else (dirs, links, deletes) keeps running -> can recover
disk space through pulling.
2018-08-25 10:16:38 +02:00
Jakob Borg
f822b10550
all: Add receive only folder type (#5027)
Adds a receive only folder type that does not send changes, and where the user can optionally revert local changes. Also changes some of the icons to make the three folder types distinguishable.
2018-07-12 11:15:57 +03:00
Simon Frei
0f0290d574 lib/model, lib/weakhash: Abort pulling quicker on folder stop (ref #5028) 2018-07-04 08:07:33 +01:00
Simon Frei
54e17c8bf4 lib/model: Don't set watch error on folder creation (fixes 5005) (#5006) 2018-06-15 23:33:23 +01:00
Simon Frei
9e0e04f4de all: Fix FS watcher restarting and web UI indication (fixes #4923) (#4962) 2018-06-11 15:47:54 +02:00
Simon Frei
b7234785f8
lib/model: Wait for folder to stop (fixes #4981) (#4982) 2018-06-10 13:24:59 +02:00
Simon Frei
9d2b744c12 lib/model: Move String method to folder (#4964) 2018-05-23 08:23:21 +01:00
Jakob Borg
119335930c lib/model: Refactor override implementation into sendOnlyFolder
I'm trying to slowly clean this up a bit, and moving functionality out
into the folder types and having those methods not reach into model is
part of it. That can mean takign some odd arguments in the meantime,
some of those should probably become interfaces or properties on folder
in the long term.
2018-05-21 09:09:47 +02:00
Jakob Borg
370a3549e7 lib/model: Refactor folderScanner into folder
The functionality was anyway mostly implemented there and not isolated
in the folderScanner type. The attempt to refactor it out in the other
direction wouldn't work given that the event loop and stuff is on
`folder`.
2018-05-21 08:47:58 +02:00
Jakob Borg
344e52e311
lib/model: Refactor {ro,rw}folder serve loop into just folder (#4939)
The actual pull method (which is really the only thing that differs
between them) is now an interface member which gets overridden by the
subclass.

"Subclass?!" Well, this is dynamic dispatch with overriding, I guess.
2018-05-11 10:45:13 +02:00
Simon Frei
158859a1e2 lib: Handle metadata changes for send-only folders (fixes #4616, fixes #4627) (#4750)
Unignored files are marked as conflicting while scanning, which is then resolved
in the subsequent pull. Automatically reconciles needed items on send-only
folders, if they do not actually differ except for internal metadata.
2018-02-25 09:39:00 +01:00
Simon Frei
f87f13081b all: Display fs watcher status and retry starting it (ref #4552) (#4713) 2018-02-04 22:46:24 +01:00
Jakob Borg
0901350087 lib/model: Trigger a pull when ignore patterns change
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4532
2017-11-20 16:29:36 +00:00
Audrius Butkevicius
386cb274bd lib/model: Incremental block stats usage reporting
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4498
LGTM: calmh
2017-11-09 21:16:29 +00:00
Simon Frei
f6ea2a7f8e lib/model: Trigger pulls instead of pulling periodically
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4340
2017-11-07 06:59:35 +00:00
Simon Frei
4b69d0e093 lib/model: Add initial noop watch cancel func (fixes #4464)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4465
2017-10-26 11:49:06 +00:00
Simon Frei
dc42db444b lib/model, lib/config: Refactor folder health/error handling (fixes #4445, fixes #4451)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4455
LGTM: AudriusButkevicius, calmh
2017-10-24 07:58:55 +00:00
Michael Ploujnikov
f98c21b68e all: Add filesystem notification support
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3986
2017-10-20 14:52:55 +00:00
Audrius Butkevicius
2760d032ca cmd/syncthing: Add more stats to usage reports (ref #3628)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4347
2017-10-12 06:16:46 +00:00
Simon Frei
f7d2c58783 lib/model: Deduplicate folder loops
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4352
LGTM: AudriusButkevicius, calmh
2017-09-07 06:17:47 +00:00
Simon Frei
ddf6d64faa lib/model: Create folders via newFolder
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4329
2017-08-25 19:47:01 +00:00
Jakob Borg
d6fbfc3545 lib/fs, lib/model, lib/scanner: Make scans cancellable (fixes #3965)
The folder already knew how to stop properly, but the fs.Walk() didn't
and can potentially take a very long time. This adds context support to
Walk and the underlying scanning stuff, and passes in an appropriate
context from above. The stop channel in model.folder is replaced with a
context for this purpose.

To test I added an infiniteFS that represents a large amount of data
(not actually infinite, but close) and verify that walking it is
properly stopped. For that to be implemented smoothly I moved out the
Walk function to it's own type, as typically the implementer of a new
filesystem type might not need or want to reimplement Walk.

It's somewhat tricky to test that this actually works properly on the
actual sendReceiveFolder and so on, as those are started from inside the
model and the filesystem isn't easily pluggable etc. Instead I've tested
that part manually by adding a huge folder and verifying that pause,
resume and reconfig do the right things by looking at debug output.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4117
2017-04-26 00:15:23 +00:00
Simon Frei
6b188ebcf3 lib/model: Mark initial scan as finished even if failed and refactor (fixes #4103)
The mechanism to disallow manual scans before the initial scan completed
(#3996) , had the side effect, that if the initial scan failed, no further
scans are allowed. So this marks the initial scan as finished regardless of
whether it succeeded or not.

There was also redundant code in rofolder and a pointless check for folder
health in scanSubsIfHealthy (happens in internalScanFolderSubdirs as well).
This also moves logging from folder.go to ro/rw-folder.go to include the
information about whether it is send-only or send-receive

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4104
2017-04-20 00:20:34 +00:00
Simon Frei
25b314f5f1 lib/model, gui: Allow creating and editing ignores of paused folders (fixes #3608)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3996
LGTM: calmh, AudriusButkevicius
2017-04-01 09:58:06 +00:00
Jakob Borg
f7fc0c1d3e all: Update license url to https (ref #3976) 2017-02-09 08:04:16 +01:00
Lars K.W. Gohlke
af0bc95de5 lib/model: Refactor encapsulation of the folder scanning
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3017
2016-06-29 06:37:34 +00:00
Lars K.W. Gohlke
236f121c4e lib/model: Refactor out folder and folderscan types, simplify somewhat
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3007
2016-04-26 14:01:46 +00:00