Commit Graph

308 Commits

Author SHA1 Message Date
Simon Frei
0db3b7a530
build: Switch to gopsutil's v3 module (#7191) 2020-12-10 16:43:15 +01:00
Simon Frei
24af89c8e2
all: Refactor preparing configuration (#7127) 2020-11-20 14:21:54 +01:00
Simon Frei
e8fc465ea8
lib/config: Use correct var in MaxConcurrentIncomingRequestKiB() (#7121) 2020-11-16 16:51:51 +01:00
Rahmi Pruitt
2f6a25a56f
gui: Add advance config port mapping to gui (fixes #4824) (#7017) 2020-11-10 09:24:11 +01:00
Simon Frei
31559e908b
all: Add untrusted folders behind feature flag (ref #62) (#7055) 2020-11-09 15:33:32 +01:00
Simon Frei
d0ccea0404
lib/config: Sanity checks on MaxConcurrentWrites (ref #7064) (#7069) 2020-11-03 19:09:32 +01:00
Tomasz Wilczyński
4a616f3cb2
lib/config: Check for "msdos" when detecting FAT FS in Android (#7072) 2020-10-30 15:13:56 +01:00
Simon Frei
f0f60ba2e7
lib/api: Add /rest/config endpoint (fixes #6540) (#7001) 2020-10-22 19:54:35 +02:00
Simon Frei
48da6f0f22
lib: Use uint64 for disk stats (ref #3930) (#7019) 2020-10-02 15:22:28 +02:00
Audrius Butkevicius
e027175446
all: Move remaining protos to use the vanity plugin (#7009) 2020-10-02 08:07:05 +02:00
Jakob Borg
7774932302
lib/config, lib/syncthing: Only drop delta index on upgrade if so ordered (fixes #6982) (#6983) 2020-09-30 20:16:30 +02:00
Audrius Butkevicius
b19b5c95d3
lib/connections: Announce LAN addresses by default (fixes #6928) (#6896) 2020-08-25 11:48:14 +02:00
Audrius Butkevicius
d507d932b8
all: Use protobuf to generate config structs (fixes #6734) (#6900) 2020-08-25 08:11:14 +02:00
Simon Frei
fc2c46e82f
lib/config, lib/fs: Make junction behaviour configurable (ref #6606) (#6907) 2020-08-19 19:58:51 +02:00
Audrius Butkevicius
bf9ff17267
all: Remove need to restart syncthing (#6883) 2020-08-18 09:26:33 +02:00
Jakob Borg
62cff26edf lib/config: Repair versioning XML serialization (fixes #6859, ref #6693) 2020-07-28 12:08:37 +02:00
Simon Frei
932d8c69de lib/fs: Properly handle case insensitive systems (fixes #1787, fixes #2739, fixes #5708)
With this change we emulate a case sensitive filesystem on top of
insensitive filesystems. This means we correctly pick up case-only renames
and throw a case conflict error when there would be multiple files differing
only in case.

This safety check has a small performance hit (about 20% more filesystem
operations when scanning for changes). The new advanced folder option
`caseSensitiveFS` can be used to disable the safety checks, retaining the
previous behavior on systems known to be fully case sensitive.

Co-authored-by: Jakob Borg <jakob@kastelo.net>
2020-07-28 11:15:11 +02:00
Jakob Borg
aedc2d788f
lib/model, lib/versioner: Drive version cleanup from scanner (fixes #6313) (#6693)
This change adds a separate config for the cleanup interval, and runs that cleanup from the main folder loop.
2020-07-14 10:48:50 +02:00
Simon Frei
8d67235a75
lib/config: Don't cache device map (fixes #6816) (#6817) 2020-07-07 23:44:49 +02:00
Audrius Butkevicius
4812fd3ec1
all: Add copy-on-write filesystem support (fixes #4271) (#6746) 2020-06-18 08:15:47 +02:00
Jonathan
9ef17322be
lib/config: Add missing quic address in case of non-default port (fixes #6679) (#6703)
* Add quic listener on instance of port blockage

* Update lib/config/config.go

Co-authored-by: Audrius Butkevicius <audrius.butkevicius@gmail.com>
2020-06-02 15:06:02 +01:00
Jędrzej Kula
28d5c84599
gui, lib/config: Add GUI user and password notification (fixes #4703) (#6536) 2020-06-02 12:08:22 +02:00
Jakob Borg
bfc9478965
cmd/syncthing: Correct auto upgrade criteria (fixes #6701) (#6702) 2020-06-02 11:38:39 +02:00
Simon Frei
b784f5b9e3
lib/config, lib/model: Commit auto-accepted folders all at once (#6684) 2020-05-27 08:05:26 +02:00
Audrius Butkevicius
6201eebc98
lib/model: Add support for different puller block ordering (#6587)
* WIP

* Tests

* Header and format

* WIP

* Fix tests

* Goland you disappoint me

* Remove CC storage

* Update blockpullreorderer.go
2020-05-11 22:44:04 +01:00
Audrius Butkevicius
782bd08aad
lib/model: Add option to disable fsync (#6588)
* lib/model: Add option to disable fsync

* Fix test

* Dont open stuff for no reason
2020-05-01 08:36:46 +01:00
Jakob Borg
8c61e0d6ab
lib/config: Sort versioning options on marshal (fixes #6576) (#6577) 2020-04-27 00:13:35 +02:00
Jakob Borg
6c73617974
lib/model: Use semaphore to limit concurrent folder writes (fixes #6541) (#6573) 2020-04-27 00:13:18 +02:00
Jakob Borg
48f9d323fa
lib/api: Add LDAP search filters (fixes #5376) (#6488)
This adds the functionality to run a user search with a filter for LDAP
authentication. The search is done after successful bind, as the binding
user. The typical use case is to limit authentication to users who are
member of a group or under a certain OU. For example, to only match
users in the "Syncthing" group in otherwise default Active Directory
set up for example.com:

    <searchBaseDN>CN=Users,DC=example,DC=com</searchBaseDN>
    <searchFilter>(&amp;(sAMAccountName=%s)(memberOf=CN=Syncthing,CN=Users,DC=example,DC=com))</searchFilter>

The search filter is an "and" of two criteria (with the ampersand being
XML quoted),

- "(sAMAccountName=%s)" matches the user logging in
- "(memberOf=CN=Syncthing,CN=Users,DC=example,DC=com)" matches members
  of the group in question.

Authentication will only proceed if the search filter matches precisely
one user.
2020-04-04 11:33:43 +02:00
Jakob Borg
80107d5f5e
lib/config: Correct spelling of address in LDAP config (#6420)
Literally noone uses this so I don't see a need to call this out or
trigger a 1.5 release for it.
2020-03-18 10:44:00 +00:00
Jakob Borg
dd92b2b8f4
all: Tweak error creation (#6391)
- In the few places where we wrap errors, use the new Go 1.13 "%w"
  construction instead of %s or %v.

- Where we create errors with constant strings, consistently use
  errors.New and not fmt.Errorf.

- Remove capitalization from errors in the few places where we had that.
2020-03-03 22:40:00 +01:00
Tyler Kropp
4026625c2d
lib/config, gui: Set unix socket permissions for GUI listen address (fixes #5979) (#6310) 2020-02-18 08:52:12 +01:00
Jakob Borg
55937b61ca
lib/model: Add global request limiter (fixes #6302) (#6303)
This adds a new config with the simple and concise name
maxConcurrentIncomingRequestKiB. This limits how many bytes we have "in
the air" in the form of response data being read and processed.

After some testing I think that not having this limiter is seldom a
great idea and thus I propose a default value of 256 MiB for this new
setting.

I also refactored the folder IO limiter to be a model/folder attribute
instead of a package global.
2020-02-01 08:02:18 +01:00
Jakob Borg
d91c4b010b
lib/config, lib/model: Limit concurrent pulls (fixes #5914) (#6290)
Adds a new folder state "Waiting to Sync" in the same vein as the
existing "Waiting to Scan". This vastly improves performances in the
rare cases when there are lots and lots of folders operating.
2020-01-27 17:31:17 +01:00
Simon Frei
e782bab9fc lib/config: Add some info to the folder marker missing (ref #5207) (#6270) 2020-01-16 15:30:29 +01:00
Simon Frei
6fd5e78740 lib: Consistently unsubscribe from config-wrapper (fixes #6133) (#6205) 2019-12-04 07:15:00 +01:00
Jakob Borg
c71116ee94
Implement database abstraction, error checking (ref #5907) (#6107)
This PR does two things, because one lead to the other:

- Move the leveldb specific stuff into a small "backend" package that
defines a backend interface and the leveldb implementation. This allows,
potentially, in the future, switching the db implementation so another
KV store should we wish to do so.

- Add proper error handling all along the way. The db and backend
packages are now errcheck clean. However, I drew the line at modifying
the FileSet API in order to keep this manageable and not continue
refactoring all of the rest of Syncthing. As such, the FileSet methods
still panic on database errors, except for the "database is closed"
error which is instead handled by silently returning as quickly as
possible, with the assumption that we're anyway "on the way out".
2019-11-29 09:11:52 +01:00
Simon Frei
b32821a586 lib/config, lib/connections: Remove ListenAddresses hack (#6188) 2019-11-26 17:07:25 +01:00
Jakob Borg
4e151d380c lib/versioner: Reduce surface area (#6186)
* lib/versioner: Reduce surface area

This is a refactor while I was anyway rooting around in the versioner.
Instead of exporting every possible implementation and the factory and
letting the caller do whatever, this now encapsulates all that and
exposes a New() that takes a config.VersioningConfiguration.

Given that and that we don't know (from the outside) how a versioner
works or what state it keeps, we now just construct it once per folder
and keep it around. Previously it was recreated for each restore
request.

* unparam

* wip
2019-11-26 07:39:31 +00:00
Simon Frei
cf312abc72 lib: Wrap errors with errors.Wrap instead of fmt.Errorf (#6181) 2019-11-23 15:20:54 +00:00
Lukas Lihotzki
96bb1c8e29 all, lib/logger: Refactor SetDebug calls (#6054) 2019-10-04 13:03:34 +02:00
ghjklw
80a83b605c lib/config: Remove stun.voxgratia.org (fixes #6010) (#6011)
DNS resolution fails for this server:
named[9495]: REFUSED unexpected RCODE resolving 'stun.voxgratia.org/A/IN': 2600:9000:5303:ae00::1#53
named[9495]: REFUSED unexpected RCODE resolving 'stun.voxgratia.org/A/IN': 205.251.198.31#53
2019-09-13 09:05:25 +01:00
Jakob Borg
90b70c7a16 lib/db: Use different defaults for larger databases (fixes #5966) (#5967)
This introduces a better set of defaults for large databases. I've
experimentally determined that it results in much better throughput in a
couple of scenarios with large databases, but I can't give any
guarantees the values are always optimal. They're probably no worse than
the defaults though.
2019-08-20 09:41:41 +02:00
Simon Frei
b1c74860e8
all: Remove global events.Default (ref #4085) (#5886) 2019-08-15 16:29:37 +02:00
Simon Frei
f6f696c6c5 lib/config: Prevent nil deref in debug logging (fixes #5955) (#5956) 2019-08-15 15:51:09 +02:00
Jakob Borg
61b9f7bd55
lib/config: Format bytes in insufficient space errors (fixes #5920) (#5921) 2019-08-02 14:43:05 +02:00
Simon Frei
05835ed81f all: Remove potentially problematic errors from panics (fixes #5839) (#5912) 2019-07-31 10:53:35 +02:00
Simon Frei
d681ac11fe
lib/config: Handle empty Fstype for mtime-window (#5906) 2019-07-30 15:23:00 +02:00
Jakob Borg
669bcb748f lib/config, lib/model: Don't save on every pending folder/device update (fixes #5888) (#5890)
Wrapper methods generally don't save by themselves.
2019-07-27 11:05:00 +01:00
Simon Frei
7a4c88d4e4 lib: Add mtime window when comparing files (#5852) 2019-07-23 21:48:53 +02:00
Simon Frei
eed1edcca0 cmd/syncthing: Ensure myID is set by making it local (fixes #5859) (#5862) 2019-07-17 07:19:14 +02:00
Simon Frei
def4b8cee5 lib/config: Error on empty folder path (fixes #5853) (#5854) 2019-07-14 11:03:14 +02:00
Jakob Borg
42ce6be9b9
lib/ur: Implement crash (panic) reporting (fixes #959) (#5702)
* lib/ur: Implement crash (panic) reporting (fixes #959)

This implements a simple crash reporting method. It piggybacks on the
panic log files created by the monitor process, picking these up and
uploading them from the usage reporting routine.

A new config value points to the crash receiver base URL, which defaults
to "https://crash.syncthing.net/newcrash" (following the pattern of
"https://data.syncthing.net/newdata" for usage reports, but allowing us
to separate the service as required).
2019-06-11 08:19:11 +02:00
Jakob Borg
997bb5e7e1 all: Remove "large blocks" config (#5763)
We now always use large / variable blocks.
2019-06-06 15:57:38 +01:00
Jakob Borg
5ce5b2c94a
lib/config: Refactor migrations a bit (#5750)
This breaks out config migrations to a separate concept, making it
(imho) slightly easier to maintain and get an overview.
2019-05-29 11:37:44 +02:00
Audrius Butkevicius
e714df013f lib/connections: Add QUIC protocol support (fixes #5377) (#5737) 2019-05-29 09:56:40 +02:00
Jakob Borg
f0f8bf7784 lib/config: Round times stored for pending folders/devices (fixes #5554) 2019-03-27 20:35:42 +01:00
Simon Frei
50d8c43e7c lib/config: Set UseLargeBlocks to true by default (fixes #5599) (#5600) 2019-03-12 12:59:26 +00:00
Simon Frei
722b3fce6a all: Hide implementations behind interfaces for mocked testing (#5548)
* lib/model: Hide implementations behind interfaces for mocked testing

* review
2019-02-26 08:09:25 +00:00
Simon Frei
d5ff2c41dc all: Get rid of fatal logging (#5537)
* cleanup Fatal in lib/config/config.go

* cleanup Fatal in lib/config/folderconfiguration.go

* cleanup Fatal in lib/model/model.go

* cleanup Fatal in cmd/syncthing/monitor.go

* cleanup Fatal in cmd/syncthing/main.go

* cleanup Fatal in lib/api

* remove Fatal methods from logger

* lowercase in errors.Wrap

* one less channel
2019-02-14 20:29:14 +00:00
Simon Frei
4299af1c63 lib/config, lib/model: Use path from locations to check disk space for db (#5525) 2019-02-12 12:25:11 +00:00
Audrius Butkevicius
dc929946fe all: Use new reflect based CLI (#5487) 2019-02-12 07:58:24 +01:00
Simon Frei
82afe73a9a cmd/syncthing, lib/config: Update default config creation (#5492)
Also remove dead code in config.Wrapper.
2019-02-02 12:43:57 +01:00
Jakob Borg
c2ddc83509 all: Revert the underscore sillyness 2019-02-02 12:16:27 +01:00
Jakob Borg
df5c1eaf01
all: Bunch of more linter fixes (#5500) 2019-02-02 11:02:28 +01:00
Simon Frei
79d109a386 lib/config: Add omitempty to DeprecatedMinHomeDiskFreePct (fixes #5482) (#5484) 2019-01-28 11:46:28 +01:00
Jakob Borg
75dcff0a0e
all: Copy owner/group from parent (fixes #5445) (#5479)
This adds a folder option "CopyOwnershipFromParent" which, when set,
makes Syncthing attempt to retain the owner/group information when
syncing files. Specifically, at the finisher stage we look at the parent
dir to get owner/group and then attempt a Lchown call on the temp file.
For this to succeed Syncthing must be running with the appropriate
permissions. On Linux this is CAP_FOWNER, which can be granted by the
service manager on startup or set on the binary in the filesystem. Other
operating systems do other things, but often it's not required to run as
full "root". On Windows this patch does nothing - ownership works
differently there and is generally less of a deal, as permissions are
inherited as ACLs anyway.

There are unit tests on the Lchown functionality, which requires the
above permissions to run. There is also a unit test on the folder which
uses the fake filesystem and hence does not need special permissions.
2019-01-25 09:52:21 +01:00
Simon Frei
643cfe2e98 lib/config: Revert #5415 (#5417)
This reverts commit 9d075781ad:
"cmd/syncthing: Improve messages when free space is running out (#5415)"
2018-12-30 21:57:41 +01:00
Maurizio Tomasi
9d075781ad cmd/syncthing: Improve messages when free space is running out (#5415) 2018-12-29 20:48:20 +01:00
Audrius Butkevicius
ff2cde469e lib/model: Allow limiting number of concurrent scans (fixes #2760) (#4888) 2018-12-05 08:40:05 +01:00
Simon Frei
4f27bdfc27 lib/model, lib/protocol: Handle request concurrency in model (#5216) 2018-11-13 08:53:55 +01:00
Simon Frei
089c283ca6 lib/config: Disable folder free disk check when configured (fixes #5267) (#5268) 2018-10-12 12:34:56 +01:00
Simon Frei
714a47ffb0 lib/config: Add context to the home disk out of space error (#5241) 2018-10-05 08:21:39 +02:00
Jakob Borg
5a69e85e80
cmd/syncthing: Listen on UNIX socket (fixes #3616) (#5210)
This adds the ability to listen on (only) a UNIX socket.
2018-09-21 14:28:57 +02:00
Audrius Butkevicius
710dba7f84 gui: Add arrays for pending/ignored folders (fixes #5190) (#5192) 2018-09-13 22:52:16 +02:00
Simon Frei
49d5eae66a lib/config: Actually modify config element (fixes #5185) (#5186) 2018-09-12 12:16:52 +01:00
Boris Rybalkin
1b1741de64 cmd/syncthing: Add LDAP authentication for GUI (fixes #5163) (#5169) 2018-09-11 23:25:24 +02:00
Audrius Butkevicius
aec66045ef
lib/config: Rewrite pending notifications (fixes #2291) 2018-08-25 11:36:10 +01:00
Simon Frei
cb0f4ce55a lib/model: Don't stop folder if out of disk space (fixes #2370) (#5099)
This removes the out of disk space check from CheckHealth. The disk space is now
only checked if there are files to pull, in which case pulling those files is
stopped, but everything else (dirs, links, deletes) keeps running -> can recover
disk space through pulling.
2018-08-25 10:16:38 +02:00
Simon Frei
ff3cbdc90d lib/model: Check availability later to catch renames (#5097) 2018-08-19 19:03:20 +01:00
Audrius Butkevicius
5161f03f02 lib/config: Fix aliased append, copy config inputs and outputs (fixes #5063) (#5069) 2018-07-26 23:14:12 +02:00
Jakob Borg
f822b10550
all: Add receive only folder type (#5027)
Adds a receive only folder type that does not send changes, and where the user can optionally revert local changes. Also changes some of the icons to make the three folder types distinguishable.
2018-07-12 11:15:57 +03:00
Simon Frei
1e2732aa21
lib/config, lib/model: Don't warn and return error (#4997) 2018-06-10 15:41:20 +02:00
Simon Frei
e2c44f519c
lib/config, lib/model: Handle shared with information in config (fixes #4870) (#4974) 2018-06-06 23:34:11 +02:00
Jakob Borg
20aa53486a
all: Serialize folder types to new names (#4942)
It's been a year and a half since we started accepting the new names.
It's time we start producing them.
2018-05-13 09:58:00 +02:00
Audrius Butkevicius
ef0dcea6a4 lib/model: Verify request content against weak (and possibly strong) hash (#4767) 2018-05-05 10:24:44 +02:00
Simon Frei
f6458d1b8f lib/config, lib/model: Include paused folders in cluster config (fixes #4897) 2018-04-22 17:01:52 +01:00
Jakob Borg
19c7cd99f5 all: Implement variable sized blocks (fixes #4807) 2018-04-16 19:08:50 +01:00
Jakob Borg
c49d864f14 lib/connections: Slightly refactor limiter juggling
Two small behavior changes: don't "charge" the data to the global rate
limit until it's been accepted by the device specific limiter, and fix
the send/recv direction in the log print on per device rate limits.
2018-03-26 11:45:02 -04:00
qepasa
2621c6fd2f lib/connections, lib/config: Bandwidth throttling per remote device (fixes #4516) (#4603) 2018-03-26 12:01:59 +02:00
Simon Frei
a557d62c4a all: Transition to using fs watcher by default (fixes #4552) 2018-03-25 21:05:47 +01:00
Simon Frei
81bd428b25 review & forgotten fixes from other PR 2018-03-18 01:10:13 +00:00
Jakob Borg
42cc64e2ed
lib/config, lib/model: Auto adjust pullers based on desired amount of pending data (#4748)
This makes the number of pullers vary with the desired amount of outstanding requests instead of being a fixed number.
2018-02-25 10:14:02 +01:00
Jakob Borg
b97d5bcca8
Remove KCP (fixes #4737) (#4741) 2018-02-09 11:40:57 +01:00
Jakob Borg
e0931e201e lib/config: Reject empty folder IDs
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4698
LGTM: imsodin
2018-01-27 09:06:37 +00:00
Jakob Borg
4afe0f407a lib/config: Verify that we reject invalid device IDs when deserializing JSON 2018-01-24 09:25:41 +01:00
Jakob Borg
c554ffccc9 cmd/syncthing, lib/config, lib/osutil: Lower process priority (fixes #4628)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4675
2018-01-15 17:11:14 +00:00
Jakob Borg
992bb0a98c lib/config, lib/discover: Support new discovery cluster (ref #4618)
This adds one new feature, that discovery servers can have ?nolookup to
be used only for announces. The default set of discovery servers is
changed to:

- discovery.s.n used for lookups. This is dual stack load balanced over
  all discovery servers, and returns both IPv4 and IPV6 results when they
  exist.

- discovery-v4.s.n used for announces. This has IPv4 addresses only and
  the discovery servers will update the unspecified address with the IPv4
  source address, as usual.

- discovery-v6.s.n which is exactly the same for IPv6.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4647
2018-01-05 14:18:32 +00:00
Audrius Butkevicius
b0e2050cdb cmd/syncthing: UI for version restoration (fixes #2599) (#4602)
cmd/syncthing: Add UI for version restoration (fixes #2599)
2018-01-01 15:39:23 +01:00
Jakob Borg
47429d01e8 lib/config, lib/model: Tweaks to the auto accept feature
Fix the folder restart behavior (ignore Label), improve the API for that
(imho).

Also removes the tab switch animation in the settings modal, because
annoying.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4577
2017-12-07 08:33:32 +00:00