Commit Graph

708 Commits

Author SHA1 Message Date
Simon Frei
08f0e125ef all: Transactionalize db.FileSet (fixes #5952) (#6239) 2020-01-21 18:23:08 +01:00
Jakob Borg
d62a0cf692
lib/model: Handle progress emitter zero interval (fixes #6281) (#6282)
Makes the logic a bit clearer and safer. This also sneakily redefines
the 0 interval to also mean disabled, whereas it previously meant ...
sometimes default to 1s, sometimes just spin.
2020-01-20 21:14:29 +01:00
Jakob Borg
6c8e8f0391
lib/model: Remove legacy handling of symlinks (#6276)
This hardly seams relevant any more; 0.14.14 is dead since a long time.
2020-01-19 12:02:20 +01:00
Simon Frei
39891cdf42 lib/model: Return paused summary instead of error on paused folders (#6272) 2020-01-17 09:30:00 +01:00
Simon Frei
e782bab9fc lib/config: Add some info to the folder marker missing (ref #5207) (#6270) 2020-01-16 15:30:29 +01:00
Simon Frei
71882765f2 lib/events, lib/model: Unflake test and prevent deadlock on event unsubscribing (#6261) 2020-01-11 08:14:05 +01:00
Simon Frei
08753ccabe
lib/model: Reset queue after all pulling is done (fixes #5867) (#6256) 2020-01-08 12:21:22 +01:00
Simon Frei
f56a5545d4 gui, lib/model: Prevent negative sync completion (fixes #4570) (#6248) 2020-01-03 14:07:57 +01:00
Jakob Borg
be0508cf26
lib/model, lib/protocol: Use error handling to avoid panic on non-started folder (fixes #6174) (#6212)
This adds error returns to model methods called by the protocol layer.
Returning an error will cause the connection to be torn down as the
message couldn't be handled. Using this to signal that a folder isn't
currently available will then cause a reconnection a few moments later,
when it'll hopefully work better.

Tested manually by running with STRECHECKDBEVERY=0 on a nontrivially
sized setup. This panics reliably before this patch, but just causes a
disconnect/reconnect now.
2019-12-04 10:46:55 +01:00
Simon Frei
6fd5e78740 lib: Consistently unsubscribe from config-wrapper (fixes #6133) (#6205) 2019-12-04 07:15:00 +01:00
Marcus Legendre
b3fd9a8d53 lib/ignore: Don't create empty ".stignore" files (fixes #6190) (#6197)
This will:

1. prevent creation of a new .stignore if there are no ignore patterns
2. delete an existing .stignore if all ignore patterns are removed
2019-12-02 08:19:02 +01:00
Jakob Borg
e82a7e3dfa
all: Propagate errors from NamespacedKV (#6203)
As foretold by the prophecy, "once the database refactor is merged, then
shall appear a request to propagate errors from the store known
throughout the land as the NamedspacedKV, and it shall be good".
2019-11-30 13:03:24 +01:00
Jakob Borg
928767e316 lib/model: gofmt lol :( 2019-11-29 09:29:59 +01:00
Jakob Borg
c71116ee94
Implement database abstraction, error checking (ref #5907) (#6107)
This PR does two things, because one lead to the other:

- Move the leveldb specific stuff into a small "backend" package that
defines a backend interface and the leveldb implementation. This allows,
potentially, in the future, switching the db implementation so another
KV store should we wish to do so.

- Add proper error handling all along the way. The db and backend
packages are now errcheck clean. However, I drew the line at modifying
the FileSet API in order to keep this manageable and not continue
refactoring all of the rest of Syncthing. As such, the FileSet methods
still panic on database errors, except for the "database is closed"
error which is instead handled by silently returning as quickly as
possible, with the assumption that we're anyway "on the way out".
2019-11-29 09:11:52 +01:00
Simon Frei
606154b183 lib/model: Also send folder summary from sync-preparing (ref #6028) (#6202) 2019-11-29 08:30:17 +01:00
Jakob Borg
4e151d380c lib/versioner: Reduce surface area (#6186)
* lib/versioner: Reduce surface area

This is a refactor while I was anyway rooting around in the versioner.
Instead of exporting every possible implementation and the factory and
letting the caller do whatever, this now encapsulates all that and
exposes a New() that takes a config.VersioningConfiguration.

Given that and that we don't know (from the outside) how a versioner
works or what state it keeps, we now just construct it once per folder
and keep it around. Previously it was recreated for each restore
request.

* unparam

* wip
2019-11-26 07:39:31 +00:00
Simon Frei
4d368a37e2 lib/model, lib/protocol: Add contexts sending indexes and download-progress (#6176) 2019-11-25 11:07:36 +01:00
Simon Frei
cf312abc72 lib: Wrap errors with errors.Wrap instead of fmt.Errorf (#6181) 2019-11-23 15:20:54 +00:00
Simon Frei
65d4dd32cb
lib/model: Also handle ServeBackground (#6173) 2019-11-22 21:30:16 +01:00
Simon Frei
57d668ed1d
lib/config: Do introductions in a single config change (#6162) 2019-11-21 08:41:41 +01:00
Simon Frei
90d85fd0a2
lib: Replace done channel with contexts in and add names to util services (#6166) 2019-11-21 08:41:15 +01:00
Simon Frei
0d14ee4142
lib/model: Don't info log repeat pull errors (#6149) 2019-11-19 09:56:53 +01:00
Simon Frei
28edf2f5bb lib/model: Keep fmut locked while adding/starting/restarting folders (#6156) 2019-11-18 21:15:26 +01:00
Simon Frei
5edf4660e2
lib/model: Prevent cleanup-race in testing (ref #6152) (#6155) 2019-11-14 23:08:40 +01:00
Simon Frei
f80ce17497
lib/model: In tests prevent goroutine leaks and increase timeouts (#6152) 2019-11-13 10:21:54 +01:00
Simon Frei
ce72bee576 lib/model: Simplify pull error/retry logic (fixes #6139) (#6141) 2019-11-11 15:50:28 +01:00
Simon Frei
85e6a77f25 lib/model: Remove some testing deadlocks (#6138) 2019-11-08 18:53:51 +01:00
Jakob Borg
88244b0c1f lib/model: Add test for previous commit 2019-11-08 17:03:25 +01:00
Simon Frei
cd290d2d05 lib/model: Add initial deviceStatRefs on model creation (fixes #6136) (#6137)
This is a regression introduced in PR #6005 / commit
f7b2e79fdc
2019-11-08 11:32:51 +00:00
Simon Frei
bee7cce081
lib/model: Add folders on start in model (#6135) 2019-11-08 10:56:16 +01:00
Ruslan Yevdokymov
3b5dff3f34 lib/model: Fix removal of a marker when there are still folders referencing it (#6114) 2019-10-30 15:11:07 +00:00
Simon Frei
bbdda059bd lib/model: Check for symlinks before deleting during pull (fixes #6090) (#6100) 2019-10-22 21:55:51 +02:00
Simon Frei
7b33294955 gui, lib/model: Add new state FolderPreparingSync (fixes #6027) (#6028) 2019-10-16 09:08:54 +02:00
Lukas Lihotzki
96bb1c8e29 all, lib/logger: Refactor SetDebug calls (#6054) 2019-10-04 13:03:34 +02:00
Audrius Butkevicius
8fb576ed54
lib/model: Adjust blocks reported in usage reporting (fixes #5995) (#6037)
* lib/model: Adjust blocks reported in usage reporting (fixes #5995)

* Use variables, fix go.mod
2019-10-04 12:03:13 +01:00
Simon Frei
a2a14c8424 lib/model: Set empty version when unignoring deleted files (fixes 6038) (#6039) 2019-10-01 15:34:59 +02:00
Simon Frei
28b6e8b063
lib/db: Update db when only local flags change (fixes #6008) (#6007) 2019-09-12 08:47:39 +02:00
Simon Frei
f7b2e79fdc lib/model: Use read-locks wherever possible (#6005) 2019-09-12 05:55:23 +01:00
Jakob Borg
ebd2e5bf30
lib/model: Correctly handle manual rescan with ignore error (fixes #5985) (#5987)
Assume a folder error was set due to bad ignores on the latest scan.
Previously, doing a manual rescan would result in:

1. Clearing the folder error, which schedules (immediately) an fs
   watcher restart

2. Attempting to load the ignores, which fails, so we set a folder
   error and bail.

3. Now the fs watcher restarts, as scheduled, so we trigger a scan.
   Goto 1.

This change fixes this by not clearing the error until the error is
actually cleared, that is, if both the health check and ignore loading
succeeds.
2019-08-30 13:27:26 +01:00
Jakob Borg
24d4290d03 lib/model, lib/scanner: Pass a valid event logger (fixes #5970) (#5971) 2019-08-21 08:05:43 +02:00
Simon Frei
77a5980747 lib/model: Do free disk space check later on pull (fixes #5948) (#5949) 2019-08-16 09:40:53 +02:00
Simon Frei
b677464dfa lib/model: Optimise locking around conn-close and puller states (#5954) 2019-08-16 09:35:19 +02:00
Simon Frei
b1c74860e8
all: Remove global events.Default (ref #4085) (#5886) 2019-08-15 16:29:37 +02:00
Simon Frei
6fa02d5081 lib/model: Fix a few more problematic locks (ref #5929) (#5944) 2019-08-13 09:04:43 +02:00
Simon Frei
86e35f1879 lib/model: Less locking in ClusterConfig (#5943) 2019-08-11 19:30:24 +02:00
Simon Frei
4a619e74f2 lib/model: Fix incorrect locking (#5939) 2019-08-11 16:10:30 +02:00
Simon Frei
8a06cf0973 lib/model: Unflake TestPullInvalidIgnored (#5918) 2019-08-01 11:07:41 +02:00
Simon Frei
05835ed81f all: Remove potentially problematic errors from panics (fixes #5839) (#5912) 2019-07-31 10:53:35 +02:00
Simon Frei
df522576ac lib/model: Don't call t.Fatal in goroutines (fixes #5901) (#5903) 2019-07-30 17:50:51 +02:00
Simon Frei
fd847d4efe lib/model: Fix flakyness of TestRequestRemoteRenameChanged (#5904) 2019-07-28 22:29:31 +02:00
Jakob Borg
c1c976aa2b
lib/model: Don't panic on failed chmod-back on directory (fixes #5836) (#5896)
* lib/model: Don't panic on failed chmod-back on directory (fixes #5836)

This makes the "in writable dir"-wrapper log chmod-back errors instead
of panicking. To do that we need a logger so the function moved into the
model package which is also the only place it's used. The tests came
along.

(The test also exercised osutil.RenameOrCopy like some sort of
piggybacking. I removed that.)
2019-07-28 10:25:05 +02:00
Jakob Borg
669bcb748f lib/config, lib/model: Don't save on every pending folder/device update (fixes #5888) (#5890)
Wrapper methods generally don't save by themselves.
2019-07-27 11:05:00 +01:00
Simon Frei
7a4c88d4e4 lib: Add mtime window when comparing files (#5852) 2019-07-23 21:48:53 +02:00
Simon Frei
35f40e9a58 lib/model: Create new file-set after stopping folder (fixes #5882) (#5883) 2019-07-23 20:39:25 +02:00
Simon Frei
6f08162376
lib/model: Remove incorrect/useless panics (#5881) 2019-07-23 10:51:16 +02:00
Simon Frei
942659fb06
lib/model, lib/nat: More service termination speedup (#5884) 2019-07-23 10:49:22 +02:00
Simon Frei
b5b54ff057 lib/model: No watch-error on missing folder (fixes #5833) (#5876) 2019-07-19 19:41:16 +02:00
Simon Frei
1cb55904bc lib/model: Prevent panic in NeedFolderFiles (fixes #5872) (#5875) 2019-07-19 19:39:52 +02:00
Simon Frei
2b622d0774 lib/model: Close conn on dev pause (fixes #5873) (#5874) 2019-07-19 19:37:29 +02:00
Simon Frei
82b70b9fae lib/model, lib/protocol: Track closing connections (fixes #5828) (#5829) 2019-07-14 11:03:55 +02:00
Simon Frei
20c8dbd9ed lib/model: Fix integer conversion (fixes #5837) (#5851) 2019-07-12 16:37:12 +02:00
Simon Frei
3446d50201 lib/model: Remove pointless error that watch hasn't started (fixes #5833) (#5834) 2019-07-10 11:00:06 +02:00
Simon Frei
9fef1552fc lib/db, lib/model: Remove folder info from panics (ref #5839) (#5840) 2019-07-10 10:57:49 +02:00
Simon Frei
ba056578ec
lib: Add util.Service as suture.Service template (fixes #5801) (#5806) 2019-07-09 11:40:30 +02:00
Simon Frei
863fe23347 gui, lib/model: Fix download progress accounting (fixes #5811) (#5815) 2019-06-30 09:23:47 +02:00
Simon Frei
1cf352a722 lib/model: NewFileSet outside fmut (#5818) 2019-06-29 08:49:30 +02:00
Simon Frei
b58f6ca886 lib/model: Correct/unify check if item changed (#5819) 2019-06-29 07:45:41 +02:00
Simon Frei
3c7e7e971d lib/model: Make jobQueue.Jobs paginated (fixes #5754) (#5804)
* lib/model: Make jobQueue.Jobs paginated (fixes #5754)

* fix, no test yet

* add test
2019-06-27 19:25:38 +01:00
Audrius Butkevicius
afde0727fe lib/versioner: Revert naming change (fixes #5807) (#5808) 2019-06-25 08:56:11 +03:00
Simon Frei
cea5962417 lib/model: Unflake TestPullInvalidIgnoredSR/SO (fixes #5796) (#5799) 2019-06-17 15:23:28 +03:00
Simon Frei
abd363e8bb
lib/model: Don't error on pulling deletion of invalid file (fixes #5791) (#5792) 2019-06-14 08:48:14 +02:00
Simon Frei
13a746e0fb lib/model: Prevent nil deref if folder stopped (fixes #5780) (#5778) 2019-06-11 11:48:51 +02:00
Audrius Butkevicius
21f50e2f8f lib/versioner: Use mtime for version cleanup (fixes #5765) (#5769) 2019-06-11 09:16:55 +02:00
Simon Frei
1054ce9354
lib/model: Refactor sending indexes as suture service (#5757) 2019-06-10 13:27:22 +02:00
Jakob Borg
997bb5e7e1 all: Remove "large blocks" config (#5763)
We now always use large / variable blocks.
2019-06-06 15:57:38 +01:00
Simon Frei
e39d3f95dd lib/protocol: Prioritize close msg and add close timeout (#5746) 2019-06-05 14:01:59 +08:00
Audrius Butkevicius
e714df013f lib/connections: Add QUIC protocol support (fixes #5377) (#5737) 2019-05-29 09:56:40 +02:00
Simon Frei
129df0613b lib/model: Unflake folder restart with blocking conn test (ref #5707) (#5744) 2019-05-27 10:58:09 +01:00
Simon Frei
486230768e lib/fs, lib/model: Add error channel to Watch to avoid panics (fixes #5697) (#5734)
* lib/fs, lib/model: Add error channel to Watch to avoid panics (fixes #5697)

* forgot unsupported watch

* and more non(-standard)-unixy fixes

* and windows test

* review
2019-05-25 20:08:26 +01:00
Simon Frei
d91da8feee lib/model: Readd special handling of conn close in TestIssue5063 (#5743)
This partially reverts commit 64518b0f7e.
2019-05-25 18:51:13 +01:00
Simon Frei
64518b0f7e
lib/model: Close connections when model is stopped (#5733) 2019-05-25 16:00:32 +02:00
Simon Frei
5b306510a0
lib/model: Consistently cleanup model in tests (#5724) 2019-05-19 14:29:07 +02:00
Simon Frei
1b2b970f32 lib/model, lib/testutils: Test closing a connection on folder restart (#5707) 2019-05-18 08:53:59 +02:00
Simon Frei
5ffbb7668d lib/model: Fix test flakyness regression (ref #5592) (#5718) 2019-05-18 08:52:50 +02:00
Simon Frei
441ea109a1
lib/model: Refactor file deletions when pulling (#5699) 2019-05-17 18:29:54 +02:00
André Colomb
e4956358fb lib/model: Remove superfluous check for IndexID in remote ClusterConfig (#5717)
The check in ClusterConfig() when iterating through announced devices
in a folder explicitly skips entries without a non-zero IndexID.
Therefore, the check for IndexID == 0 just below will never be true
and the intended cleanup of local index data will not happen.

Plainly remove that check to make the intended case distinction work.
2019-05-12 21:17:55 +02:00
Simon Frei
2b246eeb52
lib/model: Move test utilities to separate files (#5694) 2019-05-10 13:33:45 +02:00
Jakob Borg
31be810eb6 lib/model: Don't fail operation when fsync() fails (fixes #5704) (#5705) 2019-05-09 21:20:29 +01:00
Simon Frei
b45d77b6be lib/model: Fix regression deleting directories on pull (ref #5690) (#5691) 2019-05-06 20:55:26 +02:00
Simon Frei
79e67b7f79 lib/model: Remove individual pull errors on retry (fixes #5659) (#5690) 2019-05-06 17:21:56 +02:00
Simon Frei
5fa8467756 lib/model: Consistent use of locks (#5684) 2019-05-02 18:55:39 +01:00
Simon Frei
5954b105cd
lib/model: Let fakeConnection call Model.Closed on close (#5682) 2019-05-02 14:24:55 +02:00
Simon Frei
defc5dca65
lib/model: Use correct lock (#5683) 2019-05-02 14:09:42 +02:00
Simon Frei
fe4daf242b
cmd, lib/db: Actually close goleveldb (fixes #5505) (#5671) 2019-05-02 11:15:00 +02:00
Audrius Butkevicius
0ca1f26ff8
lib/versioner: Restore for all versioners, cross-device support (#5514)
* lib/versioner: Restore for all versioners, cross-device support

Fixes #4631
Fixes #4586
Fixes #1634
Fixes #5338
Fixes #5419
2019-04-28 23:30:16 +01:00
Simon Frei
5da41f75fa lib/model, lib/protocol: Wait for reader/writer loops on close (fixes #4170) (#5657)
* lib/protocol: Wait for reader/writer loops on close (fixes #4170)

* waitgroup

* lib/model: Don't hold lock while closing connection

* fix comments

* review (lock once, func argument) and naming
2019-04-28 11:58:51 +01:00
Simon Frei
110806842c lib/model: Pass the old not new fileinfo to deleteItemOnDisk (fixes #5654) (#5655) 2019-04-23 20:46:28 +01:00
Simon Frei
926e9228ed lib/scanner, lib/model: File -> item when logging error (#5664) 2019-04-21 16:19:59 +01:00
Simon Frei
86e72d9973 lib/model: Use RLock and legacy polish (#5663) 2019-04-21 13:21:36 +01:00
Simon Frei
3bea59b0d9
lib/model: Refactor progressEmitter to de-/activate by config (fixes #4613) (#5623) 2019-04-13 14:20:51 +02:00
Simon Frei
fca895a632 lib/model: Fix block index calculation for recheckFile (fixes #5649) (#5648) 2019-04-12 15:21:07 +02:00
Simon Frei
9b2a73f9ab
lib/model: Pause pull for at least as long as failed pull took (fixes #5641) (#5643) 2019-04-10 16:22:23 +02:00
Simon Frei
c305265c62 lib/model: Request errors conforming to BEP specs (#5642) 2019-04-10 11:47:24 +02:00
Simon Frei
395e524e2d lib/model: Update db on scan/pull in folder (#5608) 2019-04-07 13:29:17 +02:00
Simon Frei
8d1eff7e41 lib/model: Missing queue.Done and reset between pulls (fixes #5332) (#5626) 2019-03-27 20:19:35 +01:00
Simon Frei
0cff66fcbc lib/model: Optimize puller for meta only changes (#5622) 2019-03-27 08:36:58 +00:00
Simon Frei
43a5be1c4b lib/model: Send item finished even after deregistering (fixes #5362) (#5620) 2019-03-26 21:31:33 +01:00
Simon Frei
b50039a920 cmd/syncthing, lib/api: Separate api/gui into own package (ref #4085) (#5529)
* cmd/syncthing, lib/gui: Separate gui into own package (ref #4085)

* fix tests

* Don't use main as interface name (make old go happy)

* gui->api

* don't leak state via locations and use in-tree config

* let api (un-)subscribe to config

* interface naming and exporting

* lib/ur

* fix tests and lib/foldersummary

* shorter URVersion and ur debug fix

* review

* model.JsonCompletion(FolderCompletion) -> FolderCompletion.Map()

* rename debug facility https -> api

* folder summaries in model

* disassociate unrelated constants

* fix merge fail

* missing id assignement
2019-03-26 19:53:58 +00:00
Simon Frei
d4e81fff8a lib/model: Remove unnecessary arguments in pulling functions (#5619) 2019-03-25 12:59:22 +01:00
Simon Frei
e31a116e6e lib/model: Pass correct file info to deleteItemOnDisk (fixes #5616) (#5617) 2019-03-25 12:42:39 +01:00
Simon Frei
e7ae851900 lib/model: Debug and test fixes (#5613) 2019-03-22 14:43:47 +01:00
Simon Frei
1a6d023ba8 lib/model: Run recvonly tests in temporary dirs (#5587) 2019-03-20 17:38:03 +00:00
Simon Frei
289a02e994 lib/model: Integrate stat refs in folder (#5596) 2019-03-11 16:57:21 +00:00
Simon Frei
445637ebec lib/model: Pass fset & ignores on folder creation (#5592) 2019-03-11 07:28:54 +01:00
Simon Frei
3f3d2c814b
lib/model: Remove unused code (#5591) 2019-03-10 17:05:39 +01:00
Simon Frei
189e44488e lib/model: Introduce must test utility (#5586)
* lib/model: Introduce must test utility

* nice
2019-03-09 18:45:36 +00:00
Simon Frei
27ff20faa3 lib/model: Introduce waitForState test utility (#5585)
* lib/model: Introduce waitForState test utility

* folder id as param to waitForState
2019-03-09 10:36:55 +00:00
Simon Frei
b1564e53e4 lib/model: Improve test utilities (#5584) 2019-03-08 20:29:09 +00:00
Simon Frei
3a75b63776
lib/model: Use temp dir from osutils in tests (#5581) 2019-03-07 16:34:41 +01:00
Simon Frei
8e238c8e48
lib/model: Check before replacing existing file on pull (fixes #5571) (#5567) 2019-03-07 15:15:14 +01:00
Jakob Borg
9da3273eb8 lib/model: Clarify fileInfoBatch.flushIfFull criteria 2019-03-05 21:34:04 +01:00
Simon Frei
bd37f6da17 lib/model: Optimize dbUpdaterRoutine (#5576) 2019-03-05 21:32:37 +01:00
Simon Frei
6940d79f5b lib/model: Use errors.Wrap for pull errors (#5563) 2019-03-04 13:01:52 +00:00
Simon Frei
d6622b1f68 lib/model: setUp -> setup (#5573) 2019-03-04 12:27:10 +00:00
Simon Frei
43bcb3d5a5 lib/model: Refactor conflict name handling (#5572) 2019-03-04 12:20:40 +00:00
Jakob Borg
f24676ba5a
lib/tlsutil: Enable TLS 1.3 when available, on test builds (fixes #5065) (#5558)
* lib/tlsutil: Enable TLS 1.3 when available, on test builds (fixes #5065)

This enables TLS 1.3 negotiation on Go 1.12 by setting the GODEBUG
variable. For now, this just gets enabled on test versions (those with a
dash in the version number).

Users wishing to enable this on production builds can set GODEBUG
manually.

The string representation of connections now includes the TLS version
and cipher suite. This becomes part of the log output on connections.
That is, when talking to an old client:

    Established secure connection .../TLS1.2-TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256

and now potentially:

    Established secure connection .../TLS1.3-TLS_AES_128_GCM_SHA256

(The cipher suite was there previously in the log output, but not the
TLS version.)

I also added this info as a new Crypto() method on the connection, and
propagate this out to the API and GUI, where it can be seen in the
connection address hover (although with bad word wrapping sometimes).

* wip

* wip
2019-02-26 11:49:02 +01:00
Simon Frei
722b3fce6a all: Hide implementations behind interfaces for mocked testing (#5548)
* lib/model: Hide implementations behind interfaces for mocked testing

* review
2019-02-26 08:09:25 +00:00
Simon Frei
d5ff2c41dc all: Get rid of fatal logging (#5537)
* cleanup Fatal in lib/config/config.go

* cleanup Fatal in lib/config/folderconfiguration.go

* cleanup Fatal in lib/model/model.go

* cleanup Fatal in cmd/syncthing/monitor.go

* cleanup Fatal in cmd/syncthing/main.go

* cleanup Fatal in lib/api

* remove Fatal methods from logger

* lowercase in errors.Wrap

* one less channel
2019-02-14 20:29:14 +00:00
Simon Frei
905c3594b0 lib/model: Various model test fixes and polish (#5528)
* lib/model: Various model test fixes and polish

Missing calls to m.Stop()
Don't fail test if temporary test dir cleanup fails

* drop lazyness
2019-02-13 18:54:04 +00:00
Simon Frei
225c0dda80 lib/model: Scan conflicts after creation (#5511)
Also unflakes and improve TestRequestRemoteRenameChanged.
2019-02-12 16:05:20 +01:00
Simon Frei
5fd2cab102 lib/model: Run more tests in tmp dir (#5527) 2019-02-12 16:04:04 +01:00
Simon Frei
4299af1c63 lib/config, lib/model: Use path from locations to check disk space for db (#5525) 2019-02-12 12:25:11 +00:00
Simon Frei
d85ef949be lib/model: Introduce setupModel test utility (#5524) 2019-02-12 12:18:13 +00:00
Simon Frei
7bac927ac8 lib/model: Use functions to generate config (#5513) 2019-02-12 07:50:07 +01:00
Jakob Borg
04fdafa280 lib/db, lib/model: Remove dead code (#5517) 2019-02-08 16:42:58 +01:00
Simon Frei
2f415d8f09 lib/model: Don't use LocalDeviceID as normal id in tests (#5512) 2019-02-06 09:32:03 +01:00
Simon Frei
e538797ce1 lib/model: Improve TestIssue5063 (#5509)
* lib/model: Improve TestIssue5063

* add not that helpful issue comment
2019-02-05 23:07:21 +00:00
Simon Frei
5df8219bcb lib/model: Add progressEmitter to supervisor (model) (#5510) 2019-02-05 18:02:36 +00:00
Simon Frei
af4fb97538 lib/model: Fail test instead of panic due to closing channel twice (#5508) 2019-02-05 18:01:56 +00:00
Simon Frei
5d9d87f770 lib/model: Helperize test os and remove error return value (#5507) 2019-02-05 18:01:05 +00:00
Jakob Borg
c2ddc83509 all: Revert the underscore sillyness 2019-02-02 12:16:27 +01:00
Jakob Borg
9fd270d78e
all: A few more interesting linter fixes (#5502)
A couple of minor bugs and simplifications
2019-02-02 12:09:07 +01:00
Jakob Borg
2111386ee4
all: Fix some linter errors (#5499)
I'm working through linter complaints, these are some fixes. Broad
categories:

1) Ignore errors where we can ignore errors: add "_ = ..." construct.
you can argue that this is annoying noise, but apart from silencing the
linter it *does* serve the purpose of highlighting that an error is
being ignored. I think this is OK, because the linter highlighted some
error cases I wasn't aware of (starting CPU profiles, for example).

2) Untyped constants where we though we had set the type.

3) A real bug where we ineffectually assigned to a shadowed err.

4) Some dead code removed.

There'll be more of these, because not all packages are fixed, but the
diff was already large enough.
2019-02-02 10:11:42 +01:00
Simon Frei
7236d56731
lib/model: In tests disable watching for changes by default (fixes #5246) (#5485) 2019-01-30 16:38:10 +01:00
Jakob Borg
75dcff0a0e
all: Copy owner/group from parent (fixes #5445) (#5479)
This adds a folder option "CopyOwnershipFromParent" which, when set,
makes Syncthing attempt to retain the owner/group information when
syncing files. Specifically, at the finisher stage we look at the parent
dir to get owner/group and then attempt a Lchown call on the temp file.
For this to succeed Syncthing must be running with the appropriate
permissions. On Linux this is CAP_FOWNER, which can be granted by the
service manager on startup or set on the binary in the filesystem. Other
operating systems do other things, but often it's not required to run as
full "root". On Windows this patch does nothing - ownership works
differently there and is generally less of a deal, as permissions are
inherited as ACLs anyway.

There are unit tests on the Lchown functionality, which requires the
above permissions to run. There is also a unit test on the folder which
uses the fake filesystem and hence does not need special permissions.
2019-01-25 09:52:21 +01:00
Simon Frei
a45ba70467 lib/model: Improve errors while pulling (#5474) 2019-01-24 08:18:55 +01:00
Simon Frei
51f65bd23a lib/model: Reset pull errors when succeeding (fix #5450) (#5451) 2019-01-14 08:30:52 +01:00
Simon Frei
0b03b6a9ec lib/model: Improve filesystem operations during tests (fixes #5422)
* lib/fs, lib/model: Improve filesystem operations during tests (fixes #5422)

Introduces MustFilesystem that panics on errors and should be used for operations
during testing which must never fail.
Create temporary directories outside of testdata.

* don't do a filesystem, just a wrapper around os for testing

* fix copyright
2019-01-11 12:56:05 +00:00
Simon Frei
24ffd8be99 all: Send Close BEP msg on intentional disconnect (#5440)
This avoids waiting until next ping and timeout until the connection is actually
closed both by notifying the peer of the disconnect and by immediately closing
the local end of the connection after that. As a nice side effect, info level
logging about dropped connections now have the actual reason in it, not a generic
timeout error which looks like a real problem with the connection.
2019-01-09 17:31:09 +01:00
Jakob Borg
1e71b00936
lib/model: Sanitize paths used for auto accepted folders (fixes #5411) (#5435) 2019-01-05 18:10:02 +01:00
Simon Frei
47e08797cb lib/model: Don't pull if ignores failed to load and cleanup (#5418) 2019-01-01 10:17:14 +01:00
Simon Frei
8bb9878f26 lib/model: Check folder context before setting error state (#5416) 2018-12-30 21:56:16 +01:00
Simon Frei
4783294a09 lib/model: Don't pass nil *Matcher to performFinish (fixes #5401) (#5402) 2018-12-22 21:58:17 +01:00
Simon Frei
fc81e2b3d7 lib/model: Do folder watch operations under lock (fixes #5392) (#5395) 2018-12-21 12:06:21 +01:00
Simon Frei
a09079ed25 all: Display list of locally changed items in UI (fixes #5336) (#5337) 2018-12-11 09:59:04 +01:00
Jakob Borg
1b59960ff9
lib/model: Attempt to unflake Deregister tests (fixes #5362) (#5366) 2018-12-11 09:42:03 +01:00
Audrius Butkevicius
ff2cde469e lib/model: Allow limiting number of concurrent scans (fixes #2760) (#4888) 2018-12-05 08:40:05 +01:00
Simon Frei
2f9840ddae lib: Introduce fs.IsParent (fixes #5324) (#5326) 2018-11-22 11:16:45 +01:00
Simon Frei
db8777c29e lib/model: Add test for #5323 (#5325) 2018-11-13 08:36:16 +00:00
Simon Frei
33bed5b1ec lib/model: Don't compare permissions if IgnorePerms is true (fixes #5323) (#5322) 2018-11-13 08:54:49 +01:00
Simon Frei
4f27bdfc27 lib/model, lib/protocol: Handle request concurrency in model (#5216) 2018-11-13 08:53:55 +01:00
Simon Frei
d510e3cca3 all: Display errors while scanning in web UI (fixes #4480) (#5215) 2018-11-07 11:04:41 +01:00
Simon Frei
64a591610b lib/model: Check if files from queue are invalid (fixes #5291) (#5292) 2018-10-26 19:13:35 +01:00
Alexandre Viau
9745679c63 lib: chmod -x on progressemitter.go and errors.go (#5281) 2018-10-21 16:08:14 +01:00
Simon Frei
0690fe7585 lib/model: Unnecessary return (#5264) 2018-10-11 15:08:37 +02:00
Simon Frei
1b10607def lib/model: Don't check folder health if there is nothing to pull (fixes #2497) (#5255) 2018-10-11 11:33:21 +02:00
Jakob Borg
9d7a811e72
lib/model: Don't flake out on shortcutting files (ref #5258, #5257, #5234) (#5259)
In a recent change (#5201) this return disappeared. The effect is that
we first shortcut the file and then also treat it normally. This results
in to database updates after each other, which are bound to end up in
the same batch. This means we remove one sequence entry and add two.

Not marking the issues as fixed, because I need to do more testing and
there are other discrepancies...
2018-10-11 11:07:52 +02:00
Simon Frei
d10773c311 lib/db, lib/model: Resolve identical recv only items (fixes #5130) (#5230) 2018-10-10 12:43:07 +02:00
Simon Frei
523ac45456 lib/model: Treat failed rename like del&update (#5203) 2018-10-10 11:37:20 +02:00
Jakob Borg
b50d57b7fd
lib/db: Refactor: use a Lowlevel type underneath Instance (ref #5198) (#5212)
This adds a thin type that holds the state associated with the
leveldb.DB, leaving the huge Instance type more or less stateless. Also
moves some keying stuff into the DB package so that other packages need
not know the keying specifics.

(This does not, yet, fix the cmd/stindex program, in order to keep the
diff size down. Hence the keying constants are still exported.)
2018-10-10 11:34:24 +02:00
Jakob Borg
f12ca95af2 lib/model: Unflake TestFolderRestartZombies (fixes #5244) 2018-10-07 13:58:25 +02:00
Jakob Borg
f528923d1e lib/model, cmd/syncthing: Wait for folder restarts to complete (fixes #5233) (#5239)
* lib/model, cmd/syncthing: Wait for folder restarts to complete (fixes #5233)

This is the somewhat ugly - but on the other hand clear - fix for what
is really a somewhat thorny issue. To avoid zombie folder runners a new
mutex is introduced that protects the RestartFolder operation. I hate
adding more mutexes but the alternatives I can think of are worse.

The other part of it is that the POST /rest/system/config operation now
waits for the config commit to complete. The point of this is that until
the commit has completed we should not accept another config commit. If
we did, we could end up with two separate RestartFolders queued in the
background. While they are both correct, and will run without
interfering with each other, we can't guarantee the order in which they
will run. Thus it could happen that the newer config got committed
first, and the older config commited after that, leaving us with the
wrong config running.

* test

* wip

* hax

* hax

* unflake test

* per folder mutexes

* paranoia

* race
2018-10-05 09:26:25 +01:00
Simon Frei
c2b0d309fb lib/model: Prevent repeat db update (#5231) 2018-09-27 07:41:40 +02:00
Simon Frei
272fb3b444
all: Adjust windows perms in fs package (#5200) 2018-09-16 16:09:56 +02:00
Simon Frei
60eb9088ff lib/model: Extend shortcutFile (#5201) 2018-09-16 10:29:06 +01:00
Simon Frei
c8652222ef all: Check files on disk/in db when deleting/renaming (fixes #5194) (#5195) 2018-09-16 09:48:14 +02:00
Simon Frei
a57fa9cfab lib/model: Polish (#5189) 2018-09-13 18:17:13 +02:00
Jakob Borg
9e00b619ab all, vendor: Switch back to non-forked thejerf/suture (#5171) 2018-09-08 12:56:56 +03:00
Jakob Borg
4cb6bb6f64 Merge branch 'release'
* release:
  lib/db: Fix inconsistency in sequence index (fixes #5149) (#5158)
2018-09-02 21:05:53 +02:00
Jakob Borg
b80da29b23 lib/db: Fix inconsistency in sequence index (fixes #5149) (#5158)
The problem here is that we would update the sequence index before
updating the FileInfos, which would result in a high sequence number
pointing to a low-sequence FileInfo. The index sender would pick up the
high sequence number, send the old file, and think everything was good.
On the receiving side the old file is a no-op and ignored. The file
remains out of sync until another update for it happens.

This fixes that by correcting the order of operations in the database
update: first we remove old sequence index entries, then we update the
FileInfos (which now don't have anything pointing to them) and then we
add the sequence indexes (which the index sender can see).

The other option is to add "proper" transactions where required at the
database layer. I actually have a branch for that, but it's literally
thousands of lines of diff and I'm putting that off for another day as
this solves the problem...
2018-09-02 21:02:28 +02:00
Jakob Borg
836ca50570
lib/db: Fix inconsistency in sequence index (fixes #5149) (#5158)
The problem here is that we would update the sequence index before
updating the FileInfos, which would result in a high sequence number
pointing to a low-sequence FileInfo. The index sender would pick up the
high sequence number, send the old file, and think everything was good.
On the receiving side the old file is a no-op and ignored. The file
remains out of sync until another update for it happens.

This fixes that by correcting the order of operations in the database
update: first we remove old sequence index entries, then we update the
FileInfos (which now don't have anything pointing to them) and then we
add the sequence indexes (which the index sender can see).

The other option is to add "proper" transactions where required at the
database layer. I actually have a branch for that, but it's literally
thousands of lines of diff and I'm putting that off for another day as
this solves the problem...
2018-09-02 20:58:32 +02:00
Audrius Butkevicius
aec66045ef
lib/config: Rewrite pending notifications (fixes #2291) 2018-08-25 11:36:10 +01:00
Simon Frei
03c0537340 lib/model: Fix regressions detecting deletes/ignores (fixes #5125, fixes #5127) (#5129) 2018-08-25 10:32:35 +02:00
Jakob Borg
7dde6c7e3c lib/model: Clear out-of-space-errored files from queue (fixes #5143) 2018-08-25 10:26:10 +02:00
Simon Frei
cb0f4ce55a lib/model: Don't stop folder if out of disk space (fixes #2370) (#5099)
This removes the out of disk space check from CheckHealth. The disk space is now
only checked if there are files to pull, in which case pulling those files is
stopped, but everything else (dirs, links, deletes) keeps running -> can recover
disk space through pulling.
2018-08-25 10:16:38 +02:00
Jakob Borg
a02db70a63 lib/model: Process download progress messages for all folder types (fixes #5131) (#5139) 2018-08-21 18:49:35 +01:00
Simon Frei
165417c462 lib/model: Fixes on receive-only test setup and pulling (#5136) 2018-08-19 22:34:26 +01:00
Simon Frei
ff3cbdc90d lib/model: Check availability later to catch renames (#5097) 2018-08-19 19:03:20 +01:00
Simon Frei
9028969617
lib/model: Small fixes to test convenience functions (#5128) 2018-08-16 12:11:48 +02:00
Simon Frei
7c0798b622 lib/model: Always release the lock (#5126) 2018-08-15 16:44:59 +02:00
Simon Frei
ee42c46bd3 lib/model: Catch racy nil deref in ClusterConfig (#5106) 2018-08-15 16:44:20 +02:00
Simon Frei
885f6fcf28 lib/model: Always release the lock (#5126) 2018-08-15 16:33:03 +02:00
Simon Frei
dc57bab107 lib/model: Don't enter pulling state if we need nothing (fixes #4782) (#5118) 2018-08-13 20:39:25 +02:00
Jakob Borg
48795dba07
all: Don't let Suture capture panics (fixes #4758) (#5119)
Fork with new option.
2018-08-13 20:39:08 +02:00
Jakob Borg
b37c05c6b8
lib/model: Don't run watcher on recvonly tests (fixes #5110) (#5112) 2018-08-11 22:19:37 +02:00
Simon Frei
dfe4008607 lib/model: Catch racy nil deref in ClusterConfig (#5106) 2018-08-11 09:10:29 +02:00
Audrius Butkevicius
5161f03f02 lib/config: Fix aliased append, copy config inputs and outputs (fixes #5063) (#5069) 2018-07-26 23:14:12 +02:00
Audrius Butkevicius
9f87fd1fcf lib/model: More auto accept tests (#5014) 2018-07-15 20:26:20 +03:00
Simon Frei
5592b8b190 lib/model: Record error for unavailable files (#5066) 2018-07-14 14:09:23 +01:00
Jakob Borg
f822b10550
all: Add receive only folder type (#5027)
Adds a receive only folder type that does not send changes, and where the user can optionally revert local changes. Also changes some of the icons to make the three folder types distinguishable.
2018-07-12 11:15:57 +03:00