Commit Graph

740 Commits

Author SHA1 Message Date
Jakob Borg
e4db86836b lib/model: Locking in the request test 2016-11-30 13:11:06 +01:00
Jakob Borg
913a85c571 lib/model: Add simple file syncing test
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3772
2016-11-30 09:32:28 +00:00
Jakob Borg
ed4f6fc4b3 lib/connections, lib/model: Connection service should expose a single interface
Makes testing easier, which we'll need

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3771
2016-11-30 07:54:20 +00:00
Jakob Borg
91568a173a lib/model: Remove ineffectual assignment in test 2016-11-24 11:33:27 +01:00
Jakob Borg
8559e20237 lib/osutil: Don't chmod in atomic file creation (fixes #2472)
Instead, trust (and test) that the temp file has appropriate permissions
from the start. The only place where this changes our behavior is for
ignores which go from 0644 to 0600. I'm OK with that.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3756
2016-11-23 14:06:08 +00:00
Jakob Borg
26730eb083 lib/model: Fix test that relies on ignore reloading 2016-11-23 14:42:29 +01:00
Simon Frei
4160ce674d model: consistently use cfg when referring to config instance and not package
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3755
2016-11-22 23:14:20 +00:00
Jakob Borg
cc9a9fb390 lib/model, lib/protocol: Add Folder.Description() for logging (ref #3741) 2016-11-22 08:36:14 +01:00
Roman Zaynetdinov
d3a251e6d9 lib/model: Log folder IDs and labels (fixes #3724)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3741
2016-11-21 20:09:18 +01:00
Unrud
1574b7d834 lib/model: Add fsync of files and directories, option to disable (fixes #3711) 2016-11-21 18:09:51 +01:00
Jakob Borg
a8a0bc356a lib/model: Minor cleanup to not fondle cfg.Raw things in handleDeintroductions
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3739
2016-11-17 08:56:55 +00:00
Jakob Borg
faee1d5a8d lib/model: Fix locking around introduction handling (fixes #3737)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3738
2016-11-17 08:50:24 +00:00
Jakob Borg
3088dac33b lib/model: Clean up generateClusterConfig, fix spurious test failure by sorting 2016-11-17 07:45:45 +01:00
Audrius Butkevicius
f60b424d70 lib/config: Raw() -> RawCopy() 2016-11-13 09:29:35 +01:00
Audrius Butkevicius
a1a91d5ef4 lib/model: Introducer can remove stuff it introduced (fixes #1015)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3522
2016-11-13 09:29:33 +01:00
Jakob Borg
59f3d1445f Revert "lib/model: Introducer can remove stuff it introduced (fixes #1015)"
This reverts commit 0b88cf1d03.
2016-11-12 08:38:29 +01:00
Audrius Butkevicius
0b88cf1d03 lib/model: Introducer can remove stuff it introduced (fixes #1015)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3522
2016-11-11 15:54:25 +01:00
Jakob Borg
ae4cc94a9d lib/model: Fix locking order in Availability() (fixes #3634)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3714
2016-11-08 06:38:50 +00:00
Jakob Borg
3f9b75b7b3 Revert "lib/model: Introducer can remove stuff it introduced (fixes #1015)"
This reverts commit ec2b097313.
2016-11-08 14:27:32 +08:00
Audrius Butkevicius
ec2b097313 lib/model: Introducer can remove stuff it introduced (fixes #1015)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3522
2016-11-08 00:40:48 +08:00
Audrius Butkevicius
3418497f3d lib/sync: Log everything... 2016-11-03 21:33:33 +00:00
Audrius Butkevicius
7fba8cf759 lib/sync: Print all lockers, add holder to RWMutex 2016-10-30 00:17:25 +01:00
Jakob Borg
0296c23685 lib/protocol: Use DeviceID in protocol messages, with custom marshalling
This makes the device ID a real type that can be used in the protobuf
schema. That avoids the juggling back and forth from []byte in a bunch
of places and simplifies the code.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3695
2016-10-29 21:56:24 +00:00
MikeLund
74dd051d51 all: Update docs.s.n links to use https
Skip-check: authors

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3691
2016-10-27 17:02:19 +00:00
Audrius Butkevicius
815588daba lib/sync, lib/model: Capture locker routine ID, print locker details on deadlock 2016-10-18 21:00:01 +01:00
Jakob Borg
4e8c8d7e2c cmd/syncthing, lib/db, lib/model: Track more detailed file/dirs/links/deleted counts 2016-10-17 23:57:43 +02:00
Jakob Borg
b8a90b7eaa lib/model: Remove chatty delta index confirmation
This had its moment when making sure it worked initially, but nowadays
there is not point discussing the obvious on every folder on every
connect.
2016-10-17 09:21:58 +02:00
Nathan Morrison
d322ebd0b9 Add API service for local disk changes
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3626
LGTM: calmh, AudriusButkevicius
2016-09-28 15:54:13 +00:00
Jakob Borg
da0b7cc7f2 lib/model: Correct lock taking order in ConnectionStats (fixes #3596)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3597
2016-09-14 19:38:55 +00:00
Jakob Borg
46a143e80e lib/model: Handle deleted-then-ignored files (fixes #3502)
When files that were previously marked as deleted became ignored, we
used to do nothing at all. This changes that behavior to set the Invalid
bit (that we should rename to Ignored). This then becomes an update to
other devices that they should not trust our knowledge about the file in
question.

Read this diff without whitespace...

Tested by
- creating a bunch of files on s1
- letting them sync to s2
- shutting down s2
- deleting the files on s1 and rescanning
- adding the files to .stignore on s1 and rescanning
- starting up s2 and letting it sync
- observing the files are not deleted on s2, and it considers itself up
  to date.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3557
2016-09-02 13:23:24 +00:00
Jakob Borg
69b7f26e4c lib/model, cmd/syncthing: Also account for deleted files in folder summary events (ref #3496)
This should probably be reflected in the GUI somewhere as well...
2016-09-02 10:45:39 +02:00
Jakob Borg
5b37d0356c lib/model, gui: Correct completion percentages when there are lots of deletes (fixes #3496)
We used to consider deleted files & directories 128 bytes large. After
the delta indexes change a bug slipped in where deleted files would be
weighted according to their old non-deleted size. Both ways are
incorrect (but the latest change made it worse), as if there are more
files deleted than remaining data in the repo the needSize can be
greater than the globalSize, resulting in a negative completion
percentage.

This change makes it so that deleted items are zero bytes large, which
makes more sense. Instead we expose the number of files that we need to
delete as a separate field in the Completion() result, and hack the
percentage down to 95% complete if it was 100% complete but we need to
delete files. This latter part is sort of ugly, but necessary to give
the user some sort of feedback.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3556
2016-09-02 06:45:46 +00:00
Jakob Borg
0b1475169f lib/model: Correct virtual mtime handling (fixes #3516)
We previously set the mtime on the temp file, and then renamed it to the
real path. Unfortunately that means we'd save the real timestamp under
the under the temp name ".syncthing.foo.tmp" when the actual file that
we will look up on the next scan is "foo". This moves the Chtimes later,
ensuring that it gets recorded correctly under the right name.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3519
2016-08-16 18:22:19 +00:00
Audrius Butkevicius
6ec4fbc82b lib/model: Add minumum interval for progress emitter (fixes #3517)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3518
2016-08-16 18:22:01 +00:00
Jakob Borg
18cc7a663b lib: Remove osutil.Remove & osutil.RemoveAll (fixes #3513)
These are no longer required with Go 1.7. Change made by removing the
functions, doing a global s/osutil.Remove/os.Remove/.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3514
2016-08-16 10:01:58 +00:00
Jakob Borg
cf5febad47 build, cmd, lib: Minimum supported compiler version is Go 1.5 2016-08-15 08:37:32 +02:00
Jakob Borg
7776839c82 cmd/syncthing, gui: Improve completion calculation (fixes #3492)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3493
2016-08-12 06:41:43 +00:00
Jakob Borg
e52be3d83e lib/connections, lib/model: Refactor connection close handling (fixes #3466)
So there were some issues here. The main problem was that
model.Close(deviceID) was overloaded to mean "the connection was closed
by the protocol layer" and "i want to close this connection". That meant
it could get called twice - once *to* close the connection and then once
more when the connection *was* closed.

After this refactor there is instead a Closed(conn) method that is the
callback. I didn't need to change the parameter in the end, but I think
it's clearer what it means when it takes the connection that was closed
instead of a device ID. To close a connection, the new close(deviceID)
method is used instead, which only closes the underlying connection and
leaves the cleanup to the Closed() callback.

I also changed how we do connection switching. Instead of the connection
service calling close and then adding the connection, it just adds the
new connection. The model knows that it already has a connection and
makes sure to close and clean out that one before adding the new
connection.

To make sure to sequence this properly I added a new map of channels
that get created on connection add and closed by Closed(), so that
AddConnection() can do the close and wait for the cleanup to happen
before proceeding.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3490
2016-08-10 09:37:32 +00:00
Audrius Butkevicius
af3b6f9c83 lib/model, lib/config: Support "live" device removal, folder unsharing and folder configuration changes
Furthermore:
1. Cleans configs received, migrates them as we receive them.
2. Clears indexes of devices we no longer share the folder with

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3478
2016-08-07 16:21:59 +00:00
Jakob Borg
ea87bcefd6 lib/protocol, lib/model: Implement high precision time stamps (fixes #3305)
This adds a new nanoseconds field to the FileInfo, populates it during
scans and sets the non-truncated time in Chtimes calls.

The actual file modification time is defined as modified_s seconds +
modified_ns nanoseconds. It's expected that the modified_ns field is <=
1e9 (that is, all whole seconds should go in the modified_s field) but
not really enforced. Given that it's an int32 the timestamp can be
adjusted += ~2.9 seconds by the modified_ns field...

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3431
2016-08-06 13:05:59 +00:00
Jakob Borg
0655991a19 lib/db, lib/fs, lib/model: Introduce fs.MtimeFS, remove VirtualMtimeRepo
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3479
2016-08-05 17:45:45 +00:00
Jakob Borg
f368d2278f lib/config, lib/connections: Refactor handling of ignored devices (fixes #3470)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3471
2016-08-05 09:29:49 +00:00
Jakob Borg
1eb6db6ca8 cmd/syncthing, lib/...: Correctly handle ignores & invalid file names (fixes #3012, fixes #3457, fixes #3458)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3464
2016-08-05 07:13:52 +00:00
Audrius Butkevicius
08b5a7908f gui: Add one-off notifications that need to be acked
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3254
2016-08-02 08:07:30 +00:00
Jakob Borg
297240facf all: Rename LocalVersion to Sequence (fixes #3461)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3462
2016-07-29 19:54:24 +00:00
Jakob Borg
72026db599 lib/db, lib/model: Create temp sorting database in config dir (fixes #3449)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3454
2016-07-27 21:38:43 +00:00
Jakob Borg
aafc96f58f lib/model, lib/protocol: Sequence ClusterConfig properly (fixes #3448)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3452
2016-07-27 21:36:25 +00:00
Jakob Borg
7c7e8648ff lib/model: Trigger a puller iteration on connection (fixes #3451)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3453
2016-07-27 21:35:41 +00:00
Jakob Borg
7aaa1dd8a3 lib/scanner: Recheck file size and modification time after hashing (ref #3440)
To catch the case where the file changed. Also make sure we never let a
size-vs-blocklist mismatch slip through.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3443
2016-07-26 08:51:39 +00:00
Jakob Borg
6ed22d0885 lib/model: Stricter temporary file permissions
We could have a file to sync with permissions rw------- but we'd create
the temp file with rw-rw-rw- minus umask, usually rw-r--r--. This
potentially exposes private data while the file is being synced.

Similarly, when ignorePerms was set and we were reusing a temp files we
would set the permissions to rw-r--r-- explicitly, potentially
overriding a strict umask that would otherwise have had the file be
rw-------.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3437
2016-07-25 10:18:05 +00:00
Jakob Borg
47fa4b0a2c cmd/syncthing, lib/db, lib/model, lib/protocol: Implement delta indexes (fixes #438)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3427
2016-07-23 12:46:31 +00:00
Jakob Borg
8ab6b60778 lib/model: Sort outgoing index updates by LocalVersion
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3411
2016-07-21 17:21:15 +00:00
Jakob Borg
3176629410 cmd, lib: Fix ineffectual assignments (ineffasign) and comment spelling
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3405
2016-07-15 14:23:20 +00:00
Jakob Borg
fa0101bd60 lib/protocol, lib/discover, lib/db: Use protocol buffer serialization (fixes #3080)
This changes the BEP protocol to use protocol buffer serialization
instead of XDR, and therefore also the database format. The local
discovery protocol is also updated to be protocol buffer format.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3276
LGTM: AudriusButkevicius
2016-07-04 10:40:29 +00:00
Jakob Borg
6d357211b2 lib/config: Remove "Invalid" attribute (fixes #2471)
This contains the following behavioral changes:

 - Duplicate folder IDs is now fatal during startup
 - Invalid folder flags in the ClusterConfig is fatal for the connection
   (this will go away soon with the proto changes, as we won't have any
   unknown flags any more then)
 - Empty path is a folder error reported at runtime

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3370
2016-07-02 19:38:39 +00:00
Nicolas Braud-Santoni
8e39e2889d lib: Fix typos in connections/service.go and model/model.go
Skip-check: authors

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3366
LGTM: aviau
2016-06-30 13:42:53 +00:00
Lars K.W. Gohlke
af0bc95de5 lib/model: Refactor encapsulation of the folder scanning
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3017
2016-06-29 06:37:34 +00:00
Jakob Borg
a165838cbd lib/model: Decrease max temp filename length (fixes #3338, fixes #3355)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3356
2016-06-27 11:47:40 +00:00
Jakob Borg
b0d03d1f1c lib/config: Retain slash at end of path after expanding ~ (fixes #2782)
The various path cleaning operations done in in cleanedPath() removes
it, so we make sure it's added again at the end. This makes adding the
slash in prepare() unnecessary, but keep it anyway for display purposes
(people looking at the config).

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3342
2016-06-26 10:17:20 +00:00
Jakob Borg
9323f0faf8 lib/model: Refactor CheckFolderHealth into separate methods
While attempting to fix #2782 I thought the problem was the
CheckFolderHealth method, so I cleaned it up. That turned out not to be
the case, but I think this is better anyhow.

It also moves the "create folder and marker if the folder was empty in
the index" code to StartFolder where I think it makes better sense.

This is covered by a number of existing tests.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3343
2016-06-26 10:07:27 +00:00
Jakob Borg
f343c8ba36 lib/model, lib/scanner: Silence vet warnings
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3333
2016-06-20 21:00:39 +00:00
Jakob Borg
b779e22205 lib/model: Don't set ignore bit when it's already set
This adds a metric for "committed items" to the database instance that I
use in the test code, and a couple of tests that ensure that scans that
don't change anything also don't commit anything.

There was a case in the scanner where we set the invalid bit on files
that are ignored, even though they were already ignored and had the
invalid bit set. I had assumed this would result in an extra database
commit, but it was in fact filtered out by the Set... Anyway, I think we
can save some work on not pushing that change to the Set at all.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3298
2016-06-13 17:44:03 +00:00
Jakob Borg
d507126101 lib/protocol: Understand older/newer Hello messages (fixes #3287)
This is in preparation for future changes, but also improves the
handling when talking to pre-v0.13 clients. It breaks out the Hello
message and magic from the rest of the protocol implementation, with the
intention that this small part of the protocol will survive future
changes.

To enable this, and future testing, the new ExchangeHello function takes
an interface that can be implemented by future Hello versions and
returns a version indendent result type. It correctly detects pre-v0.13
protocols and returns a "too old" error message which gets logged to the
user at warning level:

   [I6KAH] 09:21:36 WARNING: Connecting to [...]:
     the remote device speaks an older version of the protocol (v0.12) not
     compatible with this version

Conversely, something entirely unknown will generate:

   [I6KAH] 09:40:27 WARNING: Connecting to [...]:
     the remote device speaks an unknown (newer?) version of the protocol

The intention is that in future iterations the Hello exchange will
succeed on at least one side and ExchangeHello will return the actual
data from the Hello together with ErrTooOld and an even more precise
message can be generated.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3289
2016-06-09 10:50:14 +00:00
Majed Abdulaziz
48245effdf lib/model, lib/stats: Keep track of folder's last scan time (ref #3143)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3250
LGTM: calmh, AudriusButkevicius
2016-06-02 19:26:52 +00:00
Audrius Butkevicius
92a23da3ec lib/model: Make the (?d) prefix actually work
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3205
2016-05-28 04:17:34 +00:00
Jakob Borg
9bb5988b4e lib/model: Don't deadlock when returning temp index block counts
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3194
2016-05-26 09:16:08 +00:00
Audrius Butkevicius
fc173bf679 lib/model: Fix wild completion percentages
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3188
2016-05-26 06:53:27 +00:00
Audrius Butkevicius
915e1ac7de lib/model: Handle (?d) deletes of directories (fixes #3164)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3170
2016-05-23 23:32:08 +00:00
Audrius Butkevicius
00be2bf18d lib/model: Track puller creation times (fixes #3145)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3150
2016-05-22 10:16:09 +00:00
Jakob Borg
44290a66b7 lib/model: Leave temp file in place when final rename fails (fixes #3146)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3148
2016-05-22 09:06:07 +00:00
Jakob Borg
a0ec4467fd cmd/syncthing: Emit new RemoteDownloadProgress event to track remote download progress
Without this the summary service doesn't know to recalculate completion
percentage for remote devices when DownloadProgress messages come in.
That means that completion percentage isn't updated in the GUI while
transfers of large files are ongoing. With this change, it updates
correctly.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3144
2016-05-22 07:52:08 +00:00
Jakob Borg
1dbc310c9b cmd/syncthing: Rename event LocalDiskUpdated -> LocalChangeDetected
I think this better reflects what it means. Also tweaks the verbose
format to be more like our other things and lightly refactors the code
to not have the boolean and include the folder in the event.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3121
2016-05-19 07:01:43 +00:00
Nate Morrison
86ca58e2a9 lib/model: Emit LocalDiskUpdated events on detecting local changes
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3055
2016-05-19 00:19:26 +00:00
Jakob Borg
2c1323ece6 lib/connections: Un-deprecate relaysEnabled (fixes #3074)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3098
2016-05-17 00:05:38 +00:00
Jakob Borg
6f743f3138 Revert "lib/model: Emit LocalDiskUpdated events on detecting local changes"
This reverts commit 5a7fad0bcd.
2016-05-14 10:55:24 +02:00
Nate Morrison
5a7fad0bcd lib/model: Emit LocalDiskUpdated events on detecting local changes
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3055
2016-05-14 08:37:07 +00:00
Jakob Borg
b993b41847 lib/config: Minor attribute updates
As discussed in
https://github.com/syncthing/docs/pull/169

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3082
2016-05-12 08:23:18 +00:00
Jakob Borg
21e116aa45 lib/scanner: Refactor scanner.Walk API
The old usage pattern was to create a Walker with a bunch of attributes,
then call Walk() on it and nothing else. This extracts the attributes
into a Config struct and exposes a Walk(cfg Config) method instead, as
there was no reason to expose the state-holding walker type.

Also creates a few no-op implementations of the necessary interfaces
so that we can skip nil checks and simiplify things here and there.

Definitely look at this diff without whitespace.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3060
2016-05-09 18:25:39 +00:00
Jakob Borg
1a703efa78 lib/model: Fix accounting error in rescan with multiple subs (fixes #3028)
When doing prefix scans in the database, "foo" should not be considered
a prefix of "foo2". Instead, it should match "foo" exactly and also
strings with the prefix "foo/". This is more restrictive than what the
standard leveldb prefix scan does so we add some code to enforce it.

Also exposes the initialScanCompleted on the rwfolder for testing, and
change it to be a channel (so we can wait for it from another
goroutine). Otherwise we can't be sure when the initial scan has
completed, and we need to wait for that or it might pick up changes
we're doing at an unexpected time.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3067
2016-05-09 12:56:21 +00:00
Jakob Borg
8b7b0a03eb lib/config: Don't require restart when adding folders/devices or changing listen address
The VersioningConfig change is because it defaults to nil but gets
deserialized to map[string]string{}. Now prepare() enforces a single
representation of the empty map.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3065
2016-05-09 11:30:19 +00:00
klemens
bd41e21c26 all: Correct spelling in comments
Skip-check: authors pr-build-mac

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3056
2016-05-08 10:54:22 +00:00
Audrius Butkevicius
674fc566bb lib/connections: Refactor
1. Removes separate relay lists and relay clients/services, just makes it a listen address
2. Easier plugging-in of other transports
3. Allows "hot" disabling and enabling NAT services
4. Allows "hot" listen address changes
5. Changes listen address list with a preferable "default" value just like for discovery
6. Debounces global discovery announcements as external addresses change (which it might alot upon starting)
7. Stops this whole "pick other peers relay by latency". This information is no longer available,
   but I don't think it matters as most of the time other peer only has one relay.
8. Rename ListenAddress to ListenAddresses, as well as in javascript land.
9. Stop serializing deprecated values to JSON

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/2982
2016-05-04 19:38:12 +00:00
Jakob Borg
09832abe50 lib/config: Change folder type attribute to a FolderType type
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3032
2016-05-04 11:26:36 +00:00
Audrius Butkevicius
eabd2fc936 lib/model: Use factories for creating folders
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3029
2016-05-04 10:47:33 +00:00
Audrius Butkevicius
29fa05ae05 lib/model: Discard download progress upon receiving an index update (fixes #2993)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3023
2016-05-01 06:49:29 +00:00
Audrius Butkevicius
8cf3a7aeda lib/model: Prettify tests (fixes #3014)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3016
2016-04-26 20:19:30 +00:00
Jakob Borg
b8c5cf1142 lib/model: Refactor: complete renaming of p -> f
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3015
2016-04-26 15:11:19 +00:00
Lars K.W. Gohlke
236f121c4e lib/model: Refactor out folder and folderscan types, simplify somewhat
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3007
2016-04-26 14:01:46 +00:00
Lars K.W. Gohlke
e87c1abd4e all: Clean up dead code
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3005
2016-04-22 21:15:36 +00:00
Audrius Butkevicius
4aa6ecb122 lib/model: Do not use WRONLY (ref #2584)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/2994
2016-04-22 08:12:10 +00:00
Audrius Butkevicius
ea54525a33 lib/connections: Try not to deadlock (fixes #2987)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/2991
2016-04-18 20:25:31 +00:00
Jakob Borg
b81c8d2e1b lib/model: Drop incoming updates for ignored items (fixes #1701)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/2975
2016-04-18 18:35:31 +00:00
Jakob Borg
f6953624dd lib/model: Test should pass go vet inspections
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/2977
2016-04-15 11:41:18 +00:00
AudriusButkevicius
1a5f524ae4 lib/model, lib/protocol: Implement temporary indexes (fixes #950)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/2252
2016-04-15 10:59:41 +00:00
Michael Ploujnikov
7533a61203 unifySubs: add two trivial test cases 2016-04-10 02:41:28 +00:00
Michael Ploujnikov
467d338fe4 lib/model: Scanning unknown items is OK as long as the parent is known (fixes #2915) 2016-04-09 11:25:06 +00:00
Audrius Butkevicius
4389bb037d lib/model: Add option for overwriting names on connect (fixes #2912) 2016-04-09 07:43:47 +00:00
Audrius Butkevicius
2eb8a9ef56 all: Dead code cleanup 2016-04-09 01:10:31 +00:00
Audrius Butkevicius
5a98af622d lib/ignore: Implement deletable ignores using (?d) prefix (fixes #1362) 2016-04-07 09:34:07 +00:00
Jakob Borg
cc1d122352 lib/model: Correctly detect deleted but previously ignored files as deleted 2016-04-04 11:53:55 +01:00
Jakob Borg
f5f0e46016 lib: Use bytes.Equal instead of bytes.Compare where possible 2016-03-31 15:12:46 +00:00
Michael Ploujnikov
66f480519b lib/model: Refactor out scanning method from rwfolder.Serve loop 2016-03-30 06:53:47 +00:00
Audrius Butkevicius
b4f9a55e6e protocol: Add "Hello" message at connection start, also for unauthed peers 2016-03-25 20:29:07 +00:00
Jakob Borg
aba2cc4db2 lib/model: Properly handle deleting multiple files when doing scans with subs (fixes #2851) 2016-03-18 12:16:33 +00:00
Jakob Borg
2df001fe5c lib/model: Correct handling of multiple subs when scanning (fixes #2851)
Previously the code failed in that it would return top-level plus a sub,
i.e. ["", "foo"], and it would consider "usr/lib" a prefix of
"usr/libexec" which it is not.
2016-03-18 08:28:44 +00:00
Jakob Borg
c439c543d0 tests: messagediff argument order should be expected, actual
So that the diff describes the changes that happened in actual as
compared to expected. The opposite is confusing.
2016-03-17 08:03:29 +01:00
Jakob Borg
17517bcc3d Don't show restart prompt when changing folder label (fixes #2840) 2016-03-16 12:18:21 +01:00
Lars K.W. Gohlke
16c3d39fd2 Add folder label in addition to ID (fixes #966)
An auto generated ID is suggested on folder creation to reduce conflicts with
folders created on other devices.
2016-03-11 09:48:46 +00:00
Audrius Butkevicius
a8ffde6f21 Add deps 2016-03-06 20:32:10 +00:00
Michael Ploujnikov
49601a63c8 Model.internalScanFolder: Don't ignore special .stfolder and .stignore files.
Fixes #2151.

Since Walk.walkAndHashFiles ignores .stfolder and .stignore, they will
never be found by fs.Get(protocol.LocalDeviceID, sub) in
Model.internalScanFolder. As a result, when asked to scan those subs
we end up scanning the whole folder.
2016-01-22 23:27:47 -05:00
Michael Ploujnikov
6c33188af3 Model.internalScanFolderSubs: Scan only requested subs.
This reverts the change introduced in 9b9fe0d Reduce scanning effort.
That commit caused us to automatically ignore the basename of the
specified subs and instead scan closest known root folder. For
example, in a folder that looks like:

Sync/
├── 00
│   ├── one
│   ├── three
│   └── two
├── 01
│   ├── one
│   ├── three
│   └── two
├── 02
│   ├── one
│   ├── three
│   └── two
└── one

calling '/rest/db/scan?folder=default&sub=01' called filepath.Walk on
the whole Sync/ folder instead of just the desired subfolder. This
contradicts the scan behavior promised by the documentation.

This is related to #2151.
2016-01-22 23:27:38 -05:00
Jakob Borg
11d4986517 Humanize serialization of version vectors (again) 2016-01-20 11:14:08 -08:00
Jakob Borg
6c1faa4bdb Don't require restart for usage reporting changes (fixes #2704) 2016-01-18 10:06:31 -08:00
Jakob Borg
2919b76947 The "OK" log level is silly and should not exist 2016-01-16 23:04:41 +01:00
Jakob Borg
acaf134dfe Don't crash on folder remove while pulling (fixes #2705) 2016-01-16 21:42:32 +01:00
Audrius Butkevicius
c4c6df179b Handle race within the job queue (fixes #1263) 2016-01-16 17:20:21 +00:00
Jakob Borg
8b3d75b339 Undo the hash algorithm additions; retain flag checks 2016-01-12 14:35:00 +01:00
Jakob Borg
acdddc0b79 Don't leak sendIndexes on disconnect (fixes #2589)
Adds a Closed() method on protocol.Connection and clears up
wireformatConnection a little too.
2016-01-11 17:57:25 +01:00
Jakob Borg
096b2d73cd Don't conflict copy conflict copies (fixes #2605) 2016-01-03 21:16:31 +01:00
Jakob Borg
0db80710aa Detect nonstandard hash algo and stop folder (ref #2314) 2016-01-01 20:14:31 +01:00
Audrius Butkevicius
80b9a3e00b Don't warn about failed ignores if folder unhealthy (fixes #2630) 2016-01-01 12:59:13 +00:00
Anderson Mesquita
6d94a3be05 Refactor rwfolder tests
This creates a few utility functions to avoid repetition and removes
some redundant checks.
2015-12-22 23:43:07 -05:00
Anderson Mesquita
3f94e70488 WIP: Consider tempfile when checking for free space (fixes #2598)
Checks the existing blocks that can be reused when downloading a file so
that it only requires the space corresponding to the missing blocks.
This will prevent syncthing from claiming the folder doesn't have enough
space when resuming download of large files after they have been
partially downloaded.
2015-12-21 13:36:08 -05:00
Jakob Borg
03935b2d64 Don't verify free space for files when folder MinDiskFreePct==0 (fixes #2600) 2015-12-19 13:59:08 +01:00
Lode Hoste
cc0b9e5088 Additional output on insufficient error (ref #2580) 2015-12-15 08:45:33 +01:00
Nate Morrison
86e8e5199e Filenames added to audit log in the LocalIndexUpdated event (fixes #2549) 2015-12-04 08:42:00 +01:00
buinsky
05a31c2686 Fix deleting folders on WinXP (fixes #2522) 2015-11-30 19:09:36 +03:00
Audrius Butkevicius
4474d200b0 Fix symlinks (fixes #2524) 2015-11-28 20:23:08 +00:00
Jakob Borg
d46f267663 Handle sparse files (fixes #245) 2015-11-21 17:58:09 +01:00
Jakob Borg
9cf653d673 Rename Model.clusterConfig to Model.generateClusterConfig 2015-11-17 12:10:08 +01:00
Jakob Borg
9fbdb6b305 Cancel a running scan 2015-11-13 15:30:21 +01:00
Jakob Borg
73285cadb6 Remove folder without restart (fixes #2262) 2015-11-13 13:32:52 +01:00
AudriusButkevicius
f7edd36931 Use virtualMtime when deciding if a file is up to date 2015-11-12 03:40:06 +00:00
Mike Boone
342036408e Fix typos. 2015-11-11 21:20:34 -05:00
Zillode
593632045d Correct commentary for ConnectionStats
Related to https://github.com/syncthing/syncthing-android/issues/473 and 944d9c84a0
ConnectionStats returns connection information for all devices, even when disconnected.
2015-11-09 23:58:06 +01:00
Jakob Borg
59565fd1d1 Woops 2015-11-07 11:25:00 +01:00
Jakob Borg
55592137a2 Use constructor functions for FolderConfiguration and DeviceConfiguration 2015-11-07 09:50:04 +01:00
Jakob Borg
686f91777c Don't force rescan dirs and symlinks
We can't look for changed modtime on these as we don't track the modtime
to start with.
2015-11-04 19:53:07 +00:00
Jakob Borg
2a4fc28318 We should pass around db.Instance instead of leveldb.DB
We're going to need the db.Instance to keep some state, and for that to
work we need the same one passed around everywhere. Hence this moves the
leveldb-specific file opening stuff into the db package and exports the
dbInstance type.
2015-10-31 12:35:30 +01:00
Jakob Borg
e1a8ea7dec Rescan changed files before pulling on top of them (fixes #2416) 2015-10-29 09:12:37 +01:00
Audrius Butkevicius
5b03c2d949 Remove dead code 2015-10-25 20:46:09 +00:00
Audrius Butkevicius
f629a998a0 Change errNoDevice message to something more human 2015-10-25 13:27:26 +00:00
Jakob Borg
c19eff4872 Revive remote client version in the GUI 2015-10-22 08:53:28 +02:00
Jakob Borg
c268e4ad1b Also keep GlobalSize in RAM 2015-10-21 21:58:48 +02:00
Jakob Borg
d4f81e8791 Keep LocalSize data in RAM 2015-10-21 21:58:48 +02:00
Matt Burke
c2a5e180b8 Emit warning when sync could overwrite configuration
Overwriting configuration files is likely to happen if a
user syncs their home directories across computers. In this
case, the biggest risk is that all nodes will end up with
the same certificate and thus Device ID.

When the model prepares a folder for syncing, it checks to
see if the configuration files this instance is using are
getting synced. If the are getting synced, and they aren't
getting ignored, a warning is emitted. The model is used
so that when a new folder is added dynamically, a warning
is also emitted.

This will not prevent a user from shooting themselves in
the foot, and will not cover all cases (e.g. symlinks).
It should provide _something_ for many users in this
situation to go on, though.
2015-10-20 12:22:27 -04:00
Jakob Borg
0f9fa9507e Tests must use locking to avoid race (fixes #2394) 2015-10-20 08:51:31 +02:00
Jakob Borg
90e0141ac5 Request() should return protocol errors 2015-10-19 15:14:41 +02:00
Jakob Borg
8435a8678e Don't validate requests against the database 2015-10-19 15:14:41 +02:00
Jakob Borg
6578ffe2c9 Merge pull request #2320 from AudriusButkevicius/proto
More proto changes
2015-10-18 08:47:30 +02:00
Audrius Butkevicius
afc917b582 Fix tests 2015-10-17 09:48:41 +01:00
Audrius Butkevicius
9f4cd7716e Add more information about the folders to ClusterConfig 2015-10-17 09:46:46 +01:00
Jakob Borg
910a7c619a Merge pull request #2381 from AudriusButkevicius/maxcon
Allow limiting max conflicts (fixes #2282)
2015-10-17 09:10:31 +09:00
Audrius Butkevicius
a323d85d32 Add more information about the device to ClusterConfig 2015-10-16 19:40:12 +01:00
Audrius Butkevicius
491a33de0b Move device name into the protocol messages 2015-10-16 19:40:12 +01:00
Audrius Butkevicius
752533489a Allow limiting max conflicts (fixes #2282) 2015-10-16 19:26:38 +01:00
Jakob Borg
368cd44558 Fix race conditions in model tests 2015-10-14 14:41:16 +09:00
Audrius Butkevicius
d985ed553a Make puller pause configurable 2015-10-10 11:36:09 +01:00
Jakob Borg
76af9ba53d Implement facility based logger, debugging via REST API
This implements a new debug/trace infrastructure based on a slightly
hacked up logger. Instead of the traditional "if debug { ... }" I've
rewritten the logger to have no-op Debugln and Debugf, unless debugging
has been enabled for a given "facility". The "facility" is just a
string, typically a package name.

This will be slightly slower than before; but not that much as it's
mostly a function call that returns immediately. For the cases where it
matters (the Debugln takes a hex.Dump() of something for example, and
it's not in a very occasional "if err != nil" branch) there is an
l.ShouldDebug(facility) that is fast enough to be used like the old "if
debug".

The point of all this is that we can now toggle debugging for the
various packages on and off at runtime. There's a new method
/rest/system/debug that can be POSTed a set of facilities to enable and
disable debug for, or GET from to get a list of facilities with
descriptions and their current debug status.

Similarly a /rest/system/log?since=... can grab the latest log entries,
up to 250 of them (hardcoded constant in main.go) plus the initial few.

Not implemented in this commit (but planned) is a simple debug GUI
available on /debug that shows the current log in an easily pasteable
format and has checkboxes to enable the various debug facilities.

The debug instructions to a user then becomes "visit this URL, check
these boxes, reproduce your problem, copy and paste the log". The actual
log viewer on the hypothetical /debug URL can poll regularly for new log
entries and this bypass the 250 line limit.

The existing STTRACE=foo variable is still obeyed and just sets the
start state of the system.
2015-10-03 18:09:53 +02:00
Jakob Borg
36c746bd9f Create missing directories 2015-10-01 09:43:16 +02:00
Jakob Borg
38a3bf3ada Merge pull request #2317 from simplypeachy/patch-1
Add missing parameter in disk space warning
2015-10-01 08:10:36 +02:00
Jakob Borg
be2ca0ea22 Revert "Case insensitive renames, part 1" 2015-09-30 21:40:04 +02:00
Jakob Borg
ddfebb17cf Case insensitive renames, part 1 2015-09-30 12:41:29 +02:00
Jakob Borg
a80fc1b062 Correctly report errors encountered parsing ignores (fixes #2309, fixes #2296)
An error on opening .stignore will satisfy os.IsNotExist() and not be
reported. Other errors will be reported and stop the folder, including
is-not-exist errors from #include as these are passed through fmt.Errorf.

Also fixes minor issue where we would not print cause of folder stopping
to the log.

Also fixes minor issue with capitalization of errors.
2015-09-29 18:04:18 +02:00
simplypeachy
f42f20c70c Update rwfolder.go
Revert copyright/grammar changes.
2015-09-27 09:50:54 +01:00
Jakob Borg
209bc59e8c Don't check for free space on master folders (fixes #2236)
Also clean up the logic a little, I thought it was a bit muddled.
2015-09-27 09:33:11 +02:00
simplypeachy
ed7791d824 Add missing parameter in disk space warning
Plus copyright bump, minor grammatical fix.
2015-09-26 16:46:42 +01:00
Jakob Borg
4581c57478 Fix import paths 2015-09-22 19:38:46 +02:00
Jakob Borg
1f0f5c1e23 Don't require free disk space when we might only update metadata
Instead, make sure we do the check as part of CheckFolderHealth before
pulling, and individually per file to try to not run out of space at
that stage.

(The latter is far from fool proof as we may pull lots of stuff in
parallell, but it's worth a try.)
2015-09-12 23:00:43 +02:00
Jakob Borg
446a938b06 lib/symlinks need not depend on protocol 2015-09-11 12:55:25 +02:00
Jakob Borg
e26e85b6d6 Only check pull file size if check is enabled (ref #2241) 2015-09-06 17:13:00 +02:00
Jakob Borg
2af3a92833 Allow fractional percentages (fixes #2233) 2015-09-05 12:39:15 +02:00
Jakob Borg
06f319a380 lib/stats need not depend on protocol 2015-09-04 13:23:18 +02:00
Jakob Borg
4a9997e449 lib/db need not depend on lib/config 2015-09-04 12:01:00 +02:00
Jakob Borg
02493251d5 Adjust defaults for number of hashers based on OS
https://forum.syncthing.net/t/syncthing-is-such-a-massive-resource-hog/5494/19?u=calmh
2015-09-01 10:30:35 +02:00
AudriusButkevicius
cf802dc67e Hide .stigore (fixes #2114) 2015-08-30 12:59:01 +01:00
AudriusButkevicius
94c52e3a77 Add scan percentages (fixes #1030) 2015-08-27 19:20:43 +01:00
Jakob Borg
944d9c84a0 Pause and resume devices (ref #215) 2015-08-23 22:00:21 +02:00
Jakob Borg
aec143b882 Refactor: make IntermediateConnection more like Connection 2015-08-23 08:55:32 +02:00
Jakob Borg
f691040936 Refactor: s/Basic/Direct/ on connection type 2015-08-23 08:43:33 +02:00
Audrius Butkevicius
42acf0ed60 Try harder removing the temp file 2015-08-22 14:18:19 +01:00
Audrius Butkevicius
8f2db99c86 Expose connection type and relay status in the UI 2015-08-19 21:11:55 +01:00
Audrius Butkevicius
27465353c1 Add incoming connection relay service 2015-08-19 20:57:33 +01:00
Audrius Butkevicius
bb876eac82 Connections have types 2015-08-19 20:55:29 +01:00
Jakob Borg
c7dbe18df6 Newest first should be different from oldest first (fixes #2161) 2015-08-19 09:42:52 +02:00
Jakob Borg
61a182077f Clarify and correct handling of existing files/directories when pulling
This fixes a corner case I discovered in the symlink branch, where we
unexpectedly succeed in "replacing" an entire non-empty directory tree
with a file or symlink. This happens when archiving is in use, as we
then just move the entire tree away into the archive. This is wrong as
we should just archive files and fail on non-empty dirs in all cases.

New handling first checks what the (old) thing is, and if it's a
directory or symlink just does the delete, otherwise does conflict
handling or archiving as appropriate.
2015-08-15 15:29:59 +02:00
Jakob Borg
a5345ac71e Don't get stuck at "Syncing 0%" when adding a new folder
The number of copiers and pullers is set to default at config loading
time, but the new folder configuration doesn't pass through config
loading so we start up with 0 copiers and 0 pullers and hence get stuck.
I moved the default handling to the puller itself instead. I think this
way is also cleaner as we get to keep the 0 in the config and the puller
gets to decide the defaults on it's own.
2015-08-14 10:35:51 +02:00
Jakob Borg
a2df691c7d Configurable home disk percentage, translations 2015-08-09 10:37:23 +02:00
Lode Hoste
dfaa999291 Stop folder when running out of disk space (fixes #2057)
& tweaks by calmh
2015-08-09 10:37:23 +02:00
Jakob Borg
7705a6c1f1 mv internal lib 2015-08-09 09:35:26 +02:00