Commit Graph

42 Commits

Author SHA1 Message Date
Jakob Borg
6a87aac84f
lib/db: Refactor key handling (ref #5198) (#5199)
This breaks out the key generation stuff into a separate type. It's
cleaner on its own, and it prepares for future stuff.
2018-09-18 10:41:06 +02:00
Jakob Borg
836ca50570
lib/db: Fix inconsistency in sequence index (fixes #5149) (#5158)
The problem here is that we would update the sequence index before
updating the FileInfos, which would result in a high sequence number
pointing to a low-sequence FileInfo. The index sender would pick up the
high sequence number, send the old file, and think everything was good.
On the receiving side the old file is a no-op and ignored. The file
remains out of sync until another update for it happens.

This fixes that by correcting the order of operations in the database
update: first we remove old sequence index entries, then we update the
FileInfos (which now don't have anything pointing to them) and then we
add the sequence indexes (which the index sender can see).

The other option is to add "proper" transactions where required at the
database layer. I actually have a branch for that, but it's literally
thousands of lines of diff and I'm putting that off for another day as
this solves the problem...
2018-09-02 20:58:32 +02:00
Jakob Borg
f822b10550
all: Add receive only folder type (#5027)
Adds a receive only folder type that does not send changes, and where the user can optionally revert local changes. Also changes some of the icons to make the three folder types distinguishable.
2018-07-12 11:15:57 +03:00
Simon Frei
881e923105 cmd/syncthing, lib/db: Abort execution if db version is too high (fixes #4994) (#5022) 2018-06-26 11:40:34 +02:00
Jakob Borg
b1b68ceedb
Add LocalFlags to FileInfo (#4952)
We have the invalid bit to indicate that a file isn't good. That's enough for remote devices. For ourselves, it would be good to know sometimes why the file isn't good - because it's an unsupported type, because it matches an ignore pattern, or because we detected the data is bad and we need to rescan it.

Or, and this is the main future reason for the PR, because it's a change detected on a receive only device. We will want something like the invalid flag for those changes, but marking them as invalid today means the scanner will rehash them. Hence something more fine grained is required.

This introduces a LocalFlags fields to the FileInfo where we can stash things that we care about locally. For example,

    FlagLocalUnsupported = 1 << 0 // The kind is unsupported, e.g. symlinks on Windows
    FlagLocalIgnored     = 1 << 1 // Matches local ignore patterns
    FlagLocalMustRescan  = 1 << 2 // Doesn't match content on disk, must be rechecked fully

The LocalFlags fields isn't sent over the wire; instead the Invalid attribute is calculated based on the flags at index sending time. It's on the FileInfo anyway because that's what we serialize to database etc.

The actual Invalid flag should after this just be considered when building the global state and figuring out availability for remote devices. It is not used for local file index entries.
2018-06-24 09:50:18 +02:00
Simon Frei
30056cd1ae lib/db: Move database schema migration into its own file (#4985) 2018-06-08 12:46:00 +02:00
Simon Frei
5baa432906 lib/db: Add index to track locally needed files (#4958)
To optimize WithNeed, which is called for the local device whenever an index
update is received. No tracking for remote devices to conserve db space, as
WithNeed is only queried for completion.
2018-06-02 15:08:32 +02:00
Simon Frei
d64d954721 lib/db: Fix prefixed walks (fixes #4925) (#4940) 2018-05-17 09:26:40 +02:00
Jakob Borg
fb198a0645 lib/db: Actually delete the correct sequence prefix 2018-05-09 12:06:29 +02:00
xjtdy888
506181599c lib/db: Remove all sequences related to the folder (fixes #4928) (#4929) 2018-05-09 08:57:42 +02:00
Simon Frei
a548014755 lib/db, lib/model: Add sequence->deviceKey to db for sending indexes (#4906)
Instead of walking and unmarshalling the entire db and sorting the resulting
file infos by sequence, add store device keys by sequence number in the
database. Thus only the required file infos need be unmarshalled and are already
sorted by index.
2018-05-01 23:39:15 +02:00
Simon Frei
8b4346c3ec lib/scanner, lib/fs: Don't create file infos with abs paths (fixes #4799) (#4800) 2018-03-12 13:18:59 +01:00
Jakob Borg
1471c15b29
cmd/syncthing, lib/db: Be nicer about dropping deltas on upgrade (#4798)
When dropping delta index IDs due to upgrade, only drop our local one.
Previously, when dropping all of them, we would trigger a full send in
both directions on first connect after upgrade. Then the other side
would upgrade, doing the same thing. Net effect is full index data gets
sent twice in both directions.

With this change we just drop our local ID, meaning we will send our
full index on first connect after upgrade. When the other side upgrades,
they will do the same. This is a bit less cruel.
2018-03-10 11:42:01 +01:00
Simon Frei
158859a1e2 lib: Handle metadata changes for send-only folders (fixes #4616, fixes #4627) (#4750)
Unignored files are marked as conflicting while scanning, which is then resolved
in the subsequent pull. Automatically reconciles needed items on send-only
folders, if they do not actually differ except for internal metadata.
2018-02-25 09:39:00 +01:00
Jakob Borg
d1d967f0cf lib/db: Keep folder meta data persistently in db (fixes #4400)
This keeps the data we need about sequence numbers and object counts
persistently in the database. The sizeTracker is expanded into a
metadataTracker than handled multiple folders, and the Counts struct is
made protobuf serializable. It gains a Sequence field to assist in
tracking that as well, and a collection of Counts become a CountsSet
(for serialization purposes).

The initial database scan is also a consistency check of the global
entries. This shouldn't strictly be necessary. Nonetheless I added a
created timestamp to the metadata and set a variable to compare against
that. When the time since the metadata creation is old enough, we drop
the metadata and rebuild from scratch like we used to, while also
consistency checking.

A new environment variable STCHECKDBEVERY can override this interval,
and for example be set to zero to force the check immediately.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4547
LGTM: imsodin
2017-12-14 09:51:17 +00:00
Simon Frei
ce29d3a574 all: Various debug logging improvements
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4529
2017-11-22 08:05:27 +00:00
Jakob Borg
d7d45d8092 lib/db: Refactor away the large genericReplace thing
This removes a significant, complex chunk of database code. The
"replace" operation walked both the old and new in lockstep and made the
relevant changes to make the new situation correct. But since delta
indexes we pretty much never need this - we just used replace to drop
the existing data and start over.

This makes that explicit and removes the complexity.

(This is one of those things that would be annoying to make case
insensitive, while the actual "drop and then insert" that we do is
easier.)

This is fairly well unit tested...

The one change to the tests is to cover the fact that previously replace
with something identical didn't bump the sequence number, while
obviously removing everything and re-inserting does. This is not
behavior we depend on anywhere.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4500
LGTM: imsodin, AudriusButkevicius
2017-11-12 20:20:34 +00:00
Simon Frei
c080f677cb all: Add invalid/ignored files to global list, announce to peers (fixes #623)
This lets us determine accurate completion status for remote peers when they
have ignored files.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4460
2017-11-11 19:18:17 +00:00
Simon Frei
7ba9e7c322 lib/db: Filter unchanged files when updating and polish
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4426
2017-10-24 20:05:29 +00:00
Jakob Borg
488444354b lib/db: Don't panic
So, when first implementing the database layer I added panics on every
unexpected error condition mostly to be sure to flush out bugs and
inconsistencies. Then it became sort of standard, and we don't seem to
have many bugs here any more so the panics are usually caused by things
like checksum errors on read. But it's not an optimal user experience to
crash all the time.

Here I've weeded out most of the panics, while retaining a few "can't
happen" ones like errors on marshalling and write that we really can't
recover from.

For the rest, I'm mostly treating any read error as "entry didn't
exist". This should mean we'll rescan the file and correct the info (if
scanning) or treat it as a new file and do conflict handling (when
pulling). In some cases things like our global stats may be slightly
incorrect until a restart, if a database entry goes suddenly missing
during runtime.

All in all, I think this makes us a bit more robust and friendly without
introducing too many risks for the user. If the database is truly toast,
probably many other things on the system will be toast as well...

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4118
2017-04-25 22:52:37 +00:00
Jakob Borg
f7fc0c1d3e all: Update license url to https (ref #3976) 2017-02-09 08:04:16 +01:00
Jakob Borg
c4ba580cbb all: Remove symlink support on Windows, SymlinksEnabled config
After this change,

- Symlinks on Windows are always unsupported. Sorry.

- Symlinks are always enabled on other platforms. They are just a small
  file like anything else. There is no need to special case them. If you
  don't want to sync some symlinks, ignore them.

- The protocol doesn't differentiate between different "types" of
  symlinks. If that distinction ever does become relevant the individual
  devices can figure it out by looking at the destination when they
  create the link.

It's backwards compatible in that all the old symlink types are still
understood to be symlinks, and the new SYMLINK type is equivalent to the
old SYMLINK_UNKNOWN which was always a valid way to do it.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3962
LGTM: AudriusButkevicius
2017-02-07 08:34:24 +00:00
Jakob Borg
920274bce4 lib/db: Don't panic on unknown folder in ListFolders (fixes #3584)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3869
2017-01-04 10:34:52 +00:00
Jakob Borg
47f22ff3e5 build: Enable gometalinter "unconvert" check 2016-12-21 14:53:45 +01:00
Jakob Borg
0655991a19 lib/db, lib/fs, lib/model: Introduce fs.MtimeFS, remove VirtualMtimeRepo
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3479
2016-08-05 17:45:45 +00:00
Jakob Borg
1eb6db6ca8 cmd/syncthing, lib/...: Correctly handle ignores & invalid file names (fixes #3012, fixes #3457, fixes #3458)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3464
2016-08-05 07:13:52 +00:00
Jakob Borg
72026db599 lib/db, lib/model: Create temp sorting database in config dir (fixes #3449)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3454
2016-07-27 21:38:43 +00:00
Jakob Borg
47fa4b0a2c cmd/syncthing, lib/db, lib/model, lib/protocol: Implement delta indexes (fixes #438)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3427
2016-07-23 12:46:31 +00:00
Jakob Borg
fa0101bd60 lib/protocol, lib/discover, lib/db: Use protocol buffer serialization (fixes #3080)
This changes the BEP protocol to use protocol buffer serialization
instead of XDR, and therefore also the database format. The local
discovery protocol is also updated to be protocol buffer format.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3276
LGTM: AudriusButkevicius
2016-07-04 10:40:29 +00:00
Jakob Borg
d57e6808cc lib/db: Fix alignment crash on 32 bit platforms
Fixes #3347
Fixes #3348
Fixes #3349

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3350
2016-06-26 13:40:51 +00:00
Jakob Borg
b779e22205 lib/model: Don't set ignore bit when it's already set
This adds a metric for "committed items" to the database instance that I
use in the test code, and a couple of tests that ensure that scans that
don't change anything also don't commit anything.

There was a case in the scanner where we set the invalid bit on files
that are ignored, even though they were already ignored and had the
invalid bit set. I had assumed this would result in an extra database
commit, but it was in fact filtered out by the Set... Anyway, I think we
can save some work on not pushing that change to the Set at all.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3298
2016-06-13 17:44:03 +00:00
Audrius Butkevicius
6f63909c65 lib/db,cmd/stindex: Expose VersionList and use it in stindex
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3231
2016-05-31 19:29:26 +00:00
Jakob Borg
1a703efa78 lib/model: Fix accounting error in rescan with multiple subs (fixes #3028)
When doing prefix scans in the database, "foo" should not be considered
a prefix of "foo2". Instead, it should match "foo" exactly and also
strings with the prefix "foo/". This is more restrictive than what the
standard leveldb prefix scan does so we add some code to enforce it.

Also exposes the initialScanCompleted on the rwfolder for testing, and
change it to be a channel (so we can wait for it from another
goroutine). Otherwise we can't be sure when the initial scan has
completed, and we need to wait for that or it might pick up changes
we're doing at an unexpected time.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3067
2016-05-09 12:56:21 +00:00
Jakob Borg
ccfcdf7f48 cmd/syncthing: Don't compact database at startup
This happens automatically in the background anyway, and it can take a
long time on low powered devices at an inconvenient time. We just want
to get up and running as quickly as possible.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3000
2016-04-22 07:34:11 +00:00
Jakob Borg
f5f0e46016 lib: Use bytes.Equal instead of bytes.Compare where possible 2016-03-31 15:12:46 +00:00
Jakob Borg
aba2cc4db2 lib/model: Properly handle deleting multiple files when doing scans with subs (fixes #2851) 2016-03-18 12:16:33 +00:00
Jakob Borg
e1ac740ac4 Use v2 of XDR package (actual changes) 2016-02-02 15:33:46 +01:00
Jakob Borg
ac190b2e39 Change DB label format (index folders, devices) 2016-01-03 19:32:40 +01:00
Jakob Borg
fa4b4dece1 Compact database on startup (ref #2400) 2015-11-24 13:17:30 +01:00
Jakob Borg
2a4fc28318 We should pass around db.Instance instead of leveldb.DB
We're going to need the db.Instance to keep some state, and for that to
work we need the same one passed around everywhere. Hence this moves the
leveldb-specific file opening stuff into the db package and exports the
dbInstance type.
2015-10-31 12:35:30 +01:00
Jakob Borg
faf4267c73 Refactor: the various db key functions should be instance methods 2015-10-31 11:27:04 +01:00
Jakob Borg
c4ad97136f Move leveldb instance and transactions into separate files 2015-10-29 08:07:51 +01:00