Commit Graph

45 Commits

Author SHA1 Message Date
Simon Frei
16698b12b1
lib/db: Extend set test with second remote (#6402) 2020-03-11 08:15:45 +01:00
Simon Frei
29736b1e33
lib/db: Add closeWaitGroup to allow async operation (#6317) 2020-02-11 14:31:43 +01:00
Simon Frei
08f0e125ef all: Transactionalize db.FileSet (fixes #5952) (#6239) 2020-01-21 18:23:08 +01:00
Jakob Borg
c71116ee94
Implement database abstraction, error checking (ref #5907) (#6107)
This PR does two things, because one lead to the other:

- Move the leveldb specific stuff into a small "backend" package that
defines a backend interface and the leveldb implementation. This allows,
potentially, in the future, switching the db implementation so another
KV store should we wish to do so.

- Add proper error handling all along the way. The db and backend
packages are now errcheck clean. However, I drew the line at modifying
the FileSet API in order to keep this manageable and not continue
refactoring all of the rest of Syncthing. As such, the FileSet methods
still panic on database errors, except for the "database is closed"
error which is instead handled by silently returning as quickly as
possible, with the assumption that we're anyway "on the way out".
2019-11-29 09:11:52 +01:00
Simon Frei
28b6e8b063
lib/db: Update db when only local flags change (fixes #6008) (#6007) 2019-09-12 08:47:39 +02:00
Jakob Borg
90b70c7a16 lib/db: Use different defaults for larger databases (fixes #5966) (#5967)
This introduces a better set of defaults for large databases. I've
experimentally determined that it results in much better throughput in a
couple of scenarios with large databases, but I can't give any
guarantees the values are always optimal. They're probably no worse than
the defaults though.
2019-08-20 09:41:41 +02:00
Simon Frei
7a4c88d4e4 lib: Add mtime window when comparing files (#5852) 2019-07-23 21:48:53 +02:00
Jakob Borg
9fd270d78e
all: A few more interesting linter fixes (#5502)
A couple of minor bugs and simplifications
2019-02-02 12:09:07 +01:00
Jakob Borg
1e69997ecd
lib/db: Fix iterating sequence index (fixes #5340) (#5462)
There was a problem in iterating the sequence index that could result
in missing updates. The issue is that while the index was (correctly)
iterated in a snapshot, the actual file infos were read dirty outside of
the snapshot. This fixes this by doing the reads inside the snapshot,
and also updates a couple of other places that did the same thing more
or less harmfully (I didn't investigate).

To avoid similar issues in the future I did some renaming of the
getFile* methods - the ones in a transaction are just getFile, while the
ones directly on the database are variants of getFileDirty to highlight
what's going on.
2019-01-18 11:34:18 +01:00
Jakob Borg
0fe4c01a28
lib/db: Fix broken test on Windows (#5347) 2018-12-05 08:24:14 +01:00
Jakob Borg
e89c4c053a lib/db: Simple case sensitivity test 2018-11-26 17:58:00 +01:00
Simon Frei
b1acc37c16 lib/db: Update local need on device removal (fixes #5294) (#5295) 2018-10-30 05:40:51 +01:00
Simon Frei
d10773c311 lib/db, lib/model: Resolve identical recv only items (fixes #5130) (#5230) 2018-10-10 12:43:07 +02:00
Jakob Borg
836ca50570
lib/db: Fix inconsistency in sequence index (fixes #5149) (#5158)
The problem here is that we would update the sequence index before
updating the FileInfos, which would result in a high sequence number
pointing to a low-sequence FileInfo. The index sender would pick up the
high sequence number, send the old file, and think everything was good.
On the receiving side the old file is a no-op and ignored. The file
remains out of sync until another update for it happens.

This fixes that by correcting the order of operations in the database
update: first we remove old sequence index entries, then we update the
FileInfos (which now don't have anything pointing to them) and then we
add the sequence indexes (which the index sender can see).

The other option is to add "proper" transactions where required at the
database layer. I actually have a branch for that, but it's literally
thousands of lines of diff and I'm putting that off for another day as
this solves the problem...
2018-09-02 20:58:32 +02:00
Jakob Borg
5cb4a9acf6 lib/db: Don't account remote invalid files (fixes #5089) (#5090) 2018-07-31 13:00:03 +02:00
Jakob Borg
f822b10550
all: Add receive only folder type (#5027)
Adds a receive only folder type that does not send changes, and where the user can optionally revert local changes. Also changes some of the icons to make the three folder types distinguishable.
2018-07-12 11:15:57 +03:00
Simon Frei
3f17bda786 lib/db: Catch unignored/conflicting files as needed (fixes #5053) (#5054) 2018-07-10 18:32:34 +03:00
Jakob Borg
b1b68ceedb
Add LocalFlags to FileInfo (#4952)
We have the invalid bit to indicate that a file isn't good. That's enough for remote devices. For ourselves, it would be good to know sometimes why the file isn't good - because it's an unsupported type, because it matches an ignore pattern, or because we detected the data is bad and we need to rescan it.

Or, and this is the main future reason for the PR, because it's a change detected on a receive only device. We will want something like the invalid flag for those changes, but marking them as invalid today means the scanner will rehash them. Hence something more fine grained is required.

This introduces a LocalFlags fields to the FileInfo where we can stash things that we care about locally. For example,

    FlagLocalUnsupported = 1 << 0 // The kind is unsupported, e.g. symlinks on Windows
    FlagLocalIgnored     = 1 << 1 // Matches local ignore patterns
    FlagLocalMustRescan  = 1 << 2 // Doesn't match content on disk, must be rechecked fully

The LocalFlags fields isn't sent over the wire; instead the Invalid attribute is calculated based on the flags at index sending time. It's on the FileInfo anyway because that's what we serialize to database etc.

The actual Invalid flag should after this just be considered when building the global state and figuring out availability for remote devices. It is not used for local file index entries.
2018-06-24 09:50:18 +02:00
Simon Frei
c2784d76e4 lib/db: Remove updated invalid files from need bucket (fixes #5007) (#5008) 2018-06-18 08:23:40 +02:00
Simon Frei
ef1633ac76
lib/db: Update global count when removing the previous global version (#4968) 2018-05-24 18:17:45 +02:00
Simon Frei
d64d954721 lib/db: Fix prefixed walks (fixes #4925) (#4940) 2018-05-17 09:26:40 +02:00
Simon Frei
a548014755 lib/db, lib/model: Add sequence->deviceKey to db for sending indexes (#4906)
Instead of walking and unmarshalling the entire db and sorting the resulting
file infos by sequence, add store device keys by sequence number in the
database. Thus only the required file infos need be unmarshalled and are already
sorted by index.
2018-05-01 23:39:15 +02:00
Simon Frei
4955297bf6
lib/protocol: Invalid files should always lose (#4747) 2018-02-10 19:40:57 +01:00
Simon Frei
364f61bda6 lib/db: Update global counts on invalidation (fixes #4701)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4702
2018-01-27 09:09:13 +00:00
Simon Frei
fae2ca8458 lib/db: Do not modify underlying array of argument
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4685
2018-01-18 12:40:43 +00:00
Jakob Borg
d1d967f0cf lib/db: Keep folder meta data persistently in db (fixes #4400)
This keeps the data we need about sequence numbers and object counts
persistently in the database. The sizeTracker is expanded into a
metadataTracker than handled multiple folders, and the Counts struct is
made protobuf serializable. It gains a Sequence field to assist in
tracking that as well, and a collection of Counts become a CountsSet
(for serialization purposes).

The initial database scan is also a consistency check of the global
entries. This shouldn't strictly be necessary. Nonetheless I added a
created timestamp to the metadata and set a variable to compare against
that. When the time since the metadata creation is old enough, we drop
the metadata and rebuild from scratch like we used to, while also
consistency checking.

A new environment variable STCHECKDBEVERY can override this interval,
and for example be set to zero to force the check immediately.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4547
LGTM: imsodin
2017-12-14 09:51:17 +00:00
Simon Frei
ce29d3a574 all: Various debug logging improvements
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4529
2017-11-22 08:05:27 +00:00
Jakob Borg
d7d45d8092 lib/db: Refactor away the large genericReplace thing
This removes a significant, complex chunk of database code. The
"replace" operation walked both the old and new in lockstep and made the
relevant changes to make the new situation correct. But since delta
indexes we pretty much never need this - we just used replace to drop
the existing data and start over.

This makes that explicit and removes the complexity.

(This is one of those things that would be annoying to make case
insensitive, while the actual "drop and then insert" that we do is
easier.)

This is fairly well unit tested...

The one change to the tests is to cover the fact that previously replace
with something identical didn't bump the sequence number, while
obviously removing everything and re-inserting does. This is not
behavior we depend on anywhere.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4500
LGTM: imsodin, AudriusButkevicius
2017-11-12 20:20:34 +00:00
Simon Frei
7ba9e7c322 lib/db: Filter unchanged files when updating and polish
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4426
2017-10-24 20:05:29 +00:00
Audrius Butkevicius
3d8b4a42b7 all: Convert folders to use filesystem abstraction
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4228
2017-08-19 14:36:56 +00:00
Jakob Borg
f7fc0c1d3e all: Update license url to https (ref #3976) 2017-02-09 08:04:16 +01:00
Jakob Borg
4e8c8d7e2c cmd/syncthing, lib/db, lib/model: Track more detailed file/dirs/links/deleted counts 2016-10-17 23:57:43 +02:00
Jakob Borg
297240facf all: Rename LocalVersion to Sequence (fixes #3461)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3462
2016-07-29 19:54:24 +00:00
Jakob Borg
47fa4b0a2c cmd/syncthing, lib/db, lib/model, lib/protocol: Implement delta indexes (fixes #438)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3427
2016-07-23 12:46:31 +00:00
Jakob Borg
fa0101bd60 lib/protocol, lib/discover, lib/db: Use protocol buffer serialization (fixes #3080)
This changes the BEP protocol to use protocol buffer serialization
instead of XDR, and therefore also the database format. The local
discovery protocol is also updated to be protocol buffer format.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3276
LGTM: AudriusButkevicius
2016-07-04 10:40:29 +00:00
Jakob Borg
b779e22205 lib/model: Don't set ignore bit when it's already set
This adds a metric for "committed items" to the database instance that I
use in the test code, and a couple of tests that ensure that scans that
don't change anything also don't commit anything.

There was a case in the scanner where we set the invalid bit on files
that are ignored, even though they were already ignored and had the
invalid bit set. I had assumed this would result in an extra database
commit, but it was in fact filtered out by the Set... Anyway, I think we
can save some work on not pushing that change to the Set at all.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3298
2016-06-13 17:44:03 +00:00
Jakob Borg
c439c543d0 tests: messagediff argument order should be expected, actual
So that the diff describes the changes that happened in actual as
compared to expected. The opposite is confusing.
2016-03-17 08:03:29 +01:00
Audrius Butkevicius
a8ffde6f21 Add deps 2016-03-06 20:32:10 +00:00
Jakob Borg
79680b1d5e Benchmark for single database update 2016-01-28 09:12:01 +01:00
Jakob Borg
2a4fc28318 We should pass around db.Instance instead of leveldb.DB
We're going to need the db.Instance to keep some state, and for that to
work we need the same one passed around everywhere. Hence this moves the
leveldb-specific file opening stuff into the db package and exports the
dbInstance type.
2015-10-31 12:35:30 +01:00
Jakob Borg
c1591a5efd Only run benchmarks with -tags benchmark
Avoids creating temp database and stuff on a normal test run
2015-10-21 23:19:26 +02:00
Jakob Borg
943e80e26c Make benchmarks more realistic 2015-10-21 23:04:29 +02:00
Jakob Borg
1eca4170f7 Add test for LocalSize/GlobalSize results 2015-10-21 21:58:48 +02:00
Jakob Borg
4581c57478 Fix import paths 2015-09-22 19:38:46 +02:00
Jakob Borg
7705a6c1f1 mv internal lib 2015-08-09 09:35:26 +02:00