Commit Graph

321 Commits

Author SHA1 Message Date
Jakob Borg
55d5e03639
lib/db: Remove Badger experiment (#7413) 2021-03-01 09:16:08 +01:00
Simon Frei
80fc238bec
all: Automatic/disabled folder-config when receive-encrypted (#7327) 2021-02-12 22:51:29 +01:00
Simon Frei
c0f353c0e8
lib: Do not set ModifiedBy on meta only changes (#7345) 2021-02-08 15:30:39 +01:00
Simon Frei
46bbc78e82
lib/db: Fix and improve removing entries from global (ref #6501) (#7336) 2021-02-08 08:38:41 +01:00
Simon Frei
27a34609a1
all: Failure reporting fixes (#7331) 2021-02-05 11:21:14 +01:00
Simon Frei
ffc14a77c6
all: Add configurable defaults (fixes #4224, fixes #6086) (#7131) 2021-02-04 21:10:41 +01:00
Simon Frei
070bf3b776
lib/db: Report number of repaired items from checkGlobal (#7329) 2021-02-04 14:42:46 +01:00
André Colomb
e32a516b5f
lib/model: Forget pending folders no longer announced in ClusterConfig (fixes #5187) (#7205)
* lib/db: Add ExpirePendingFolders().

Use-case is to drop any no-longer-pending folders for a specific
device when parsing its ClusterConfig message where previously offered
folders are not mentioned any more.

The timestamp in ObservedFolder is stored with only second precision,
so round to seconds here as well.  This allows calling the function
within the same second of adding or updating entries.

* lib/model: Weed out pending folders when receiving ClusterConfig.

Filter the entries by timestamp, which must be newer than or equal to
the reception time of the ClusterConfig.  For just mentioned ones,
this assumption will hold as AddOrUpdatePendingFolder() updates the
timestamp.

* lib/model, gui: Notify when one or more pending folders expired.

Introduce new event type FolderOfferCancelled and use it to trigger a
complete refreshCluster() cycle.  Listing individual entries would be
much more code and probably just as much work to answer the API
request.

* lib/model: Add comment and rename ExpirePendingFolders().

* lib/events: Rename FolderOfferCancelled to ClusterPendingChanged.

* lib/model: Reuse ClusterPendingChanged event for cleanPending()

Changing the config does not necessarily mean that the
/resut/cluster/pending endpoints need to be refreshed, but only if
something was actually removed.  Detect this and indicate it through
the ClusterPendingChanged event, which is already hooked up to requery
respective endpoints within the GUI.

No more need for a separate refreshCluster() in reaction to
ConfigSaved event or calling refreshConfig().

* lib/model: Gofmt.

* lib/db: Warn instead of info log for failed removal.

* gui: Fix pending notifications not loading on GUI start.

* lib/db: Use short device ID in log message.

* lib/db: Return list of expired folder IDs after deleting them.

* lib/model: Refactor Pending...Changed events.

* lib/model: Adjust format of removed pending folders enumeration.

Use an array of objects with device / folder ID properties, matching
the other places where it's used.

* lib/db: Drop invalid entries in RemovePendingFoldersBeforeTime().

* lib/model: Gofmt.

My local gofmt did not complain here, strangely...

* gui: Handle PendingDevicesChanged event.

Even though it currently only holds one device at a time, wrap the
contents in an array under the "added" property name.

* lib/model: Fix null values in PendingFoldersChanged removed member.

* gui: Handle PendingFoldersChanged event.

* lib/model: Simplify construction of expiredPendingList.

* lib/model: Reduce code duplication in cleanPending().

Use goto and a label for the common parts of calling the DB removal
function and building the event data part.

* lib/events, gui: Mark ...Rejected events deprecated.

Extend comments explaining the conditions when the replacement event
types are emitted.

* lib/model: Wrap removed devices in array of objects as well.

* lib/db: Use iter.Value() instead of needless db.Get(iter.Key())

* lib/db: Add comment explaining RemovePendingFoldersBeforeTime().

* lib/model: Rename fields folderID and deviceID in event data.

* lib/db: Only list actually expired IDs as removed.

Skip entries where Delete() failed as well as invalid entries that got
removed automatically.

* lib/model: Gofmt
2021-01-25 10:58:10 +00:00
Jakob Borg
0b193b76c2
lib/config, lib/connections: Add optional connection limits (fixes #7176) (#7223)
This adds two new configuration options:

    // The number of connections at which we stop trying to connect to more
    // devices, zero meaning no limit. Does not affect incoming connections.
    ConnectionLimitEnough int

    // The maximum number of connections which we will allow in total, zero
    // meaning no limit. Affects incoming connections and prevents
    // attempting outgoing connections.
    ConnectionLimitMax int

These can be used to limit the number of concurrent connections in
various ways.
2021-01-11 15:14:44 +01:00
Simon Frei
a20a5f61f0
lib/ur: Send unreported failures on shutdown (#7164) 2020-12-22 20:17:14 +01:00
Simon Frei
a744dee94c
lib/fs: Correct wrapping order for meaningful log-caller (#7209) 2020-12-21 13:01:34 +01:00
Simon Frei
78bd0341a8
all: Handle errors opening db/creating file-set (ref #5907) (#7150) 2020-12-21 12:59:22 +01:00
Simon Frei
4a787986cd
lib/db: Prevent IndexID creation race (#7211) 2020-12-21 11:32:59 +01:00
Simon Frei
bd0c9913cf
lib/db: Remove index ids when dropping folder (#7200) 2020-12-21 11:10:59 +01:00
Simon Frei
fa40ccece1
lib: Consistently set suture logging (#7202) 2020-12-18 19:44:00 +01:00
André Colomb
7502997e7e
all: Store pending devices and folders in database (fixes #7178) (#6443) 2020-12-17 19:54:31 +01:00
Simon Frei
a9764fc16c
lib: Skip deleted, locally changed on recv-enc folders (fixes #7153) (#7154) 2020-11-27 11:26:36 +01:00
Simon Frei
3169212046
lib/db: Do not reset index-id when dropping device (ref #7135) (#7156)
This reverts commit 641b7aee38.
2020-11-25 22:54:05 +01:00
Simon Frei
8ebd893349
lib/db: Store versions for last successful db migration (#7140) 2020-11-23 18:31:32 +01:00
Simon Frei
641b7aee38
lib/db: Reset index-id when dropping device (#7135) 2020-11-20 14:17:09 +01:00
Simon Frei
9524b51708
all: Implement suture v4-api (#6947) 2020-11-17 13:19:04 +01:00
Simon Frei
1d3b9876f6
lib/api, lib/db: Add file debug endpoint (#7095) 2020-11-10 09:24:45 +01:00
Simon Frei
31559e908b
all: Add untrusted folders behind feature flag (ref #62) (#7055) 2020-11-09 15:33:32 +01:00
Jakob Borg
7892547873
lib: Remove USE_BADGER experiment (#7089)
This removes the switch for using a Badger database, because it has bugs
that it seems there is no interest in fixing, and no actual bug tracker
to track them in.

It retains the actual implementation for the sole purpose of being able
to do the conversion back to LevelDB if anyone is actually running with
USE_BADGER. At some point in a couple of versions we can remove the
implementation as well.
2020-11-03 09:10:35 +01:00
Simon Frei
2ba3be5e4d
lib/db: Add mechanism to repair db without schema update (ref #7044) (#7047) 2020-10-21 14:21:09 +02:00
Simon Frei
27c91c57d5
lib/db: Remove need for the right dev removing globals (fixes #7036) (#7044) 2020-10-21 08:26:10 +02:00
Simon Frei
01a7ef3b0f
lib/db: Undo adding user info to panic msgs (ref #7029) (#7040) 2020-10-19 08:40:37 +02:00
Simon Frei
23c935b05a
lib/db: Ignore not found on delete in recalcGlobal (ref #7026) (#7041) 2020-10-19 08:28:53 +02:00
Simon Frei
d828adb648
cmd/stcrashreceiver, lib/db: Improve panic message handling (#7029) 2020-10-08 17:37:45 +02:00
Simon Frei
14ae330eff
lib/db: Improve error handling checking db (ref #7026) (#7027) 2020-10-06 20:14:09 +02:00
Audrius Butkevicius
e027175446
all: Move remaining protos to use the vanity plugin (#7009) 2020-10-02 08:07:05 +02:00
Simon Frei
08bebbe59b
lib/db, lib/syncthing: Don't repair DB on upgrade, but on error (fixes #6917) (#6971) 2020-09-10 10:54:41 +02:00
Simon Frei
ecc24428ac
lib/model: Prevent duplicate need count on recalculating metadata (#6964) 2020-09-07 20:19:03 +02:00
Jakob Borg
674fca3868
lib/db, lib/protocol: Never need empty-version entries (fixes #6961) (#6962)
Avoid havoc when discovering locally-deleted-upgraded files during repair / need calculation...

Co-authored-by: Simon Frei <freisim93@gmail.com>
2020-09-07 20:18:25 +02:00
Jakob Borg
415adfbae6
lib/db: Don't hang on double iterator release with Badger (#6960)
Since iterators must be released before committing or discarding a
transaction we have the pattern of both deferring a release plus doing
it manually. But we can't release twice because we track this with a
WaitGroup that will panic when there are more Done()s than Add()s. This
just adds a boolean to let an iterator keep track.
2020-09-07 10:31:33 +02:00
Simon Frei
52d1681fe6
lib/db: Copy returned, old FileVersion (#6952) 2020-09-04 14:13:38 +02:00
Simon Frei
7b821d2550
lib/db: Add local need check&repair (#6950) 2020-09-04 14:01:46 +02:00
Simon Frei
0e3e0a7c9e
cmd/stindex, lib/db: Use same condition to check for needed files (#6954) 2020-09-04 13:59:04 +02:00
Jakob Borg
71bfad0bc6 cmd/stindex: Add dumping folder meta 2020-09-04 11:39:21 +02:00
Jakob Borg
b7986801cd cmd/stindex: Print more hashes and versions 2020-09-04 09:10:33 +02:00
Audrius Butkevicius
d507d932b8
all: Use protobuf to generate config structs (fixes #6734) (#6900) 2020-08-25 08:11:14 +02:00
Simon Frei
cc1f6e4d4a
lib/db, lib/model: Cover exec-paths with debug logging (#6918) 2020-08-20 16:11:20 +02:00
Audrius Butkevicius
cbbc262161
lib/db: Dont recurse on flush (fixes #6905) (#6906)
Or else we crash.

Co-authored-by: Jakob Borg <jakob@kastelo.net>
Co-authored-by: Simon Frei <freisim93@gmail.com>
2020-08-19 13:13:44 +02:00
Simon Frei
8f5215878b
lib/db: Don't put truncated files (ref #6855, ref #6501) (#6888) 2020-08-18 09:20:12 +02:00
Simon Frei
bbf25e2d02
lib/db: Log context on panic (#6872) 2020-08-01 17:32:36 +02:00
Simon Frei
850dd4cd25
lib/db: Don't count invalid items to global state (ref #6850) (#6863) 2020-07-30 13:49:14 +02:00
Simon Frei
424d1b1608
lib/db: Commit meta when dropping device (#6862) 2020-07-28 16:46:42 +02:00
Simon Frei
1b9e5c0937
lib/db: Include blocks in db check (ref #6855) (#6861) 2020-07-28 16:25:07 +02:00
Simon Frei
08e0f938a9
lib/db: Update global meta even if unchanged (fixes #6850) (#6852) 2020-07-24 12:36:16 +02:00
Audrius Butkevicius
55147f5901
lib/db: Rework flush hooks (#6838) 2020-07-19 08:55:27 +02:00
greatroar
9f92f8c609
lib/db: Use SipHash to deal with hash collision in GC (#6826)
If the GC finds a key k that it wants to keep, it records that in a
Bloom filter. If a key k' can be removed but its hash collides with k,
it will be kept. Since the old Bloom filter code was completely
deterministic, the next run would encounter the same collision, assuming
k must still be kept.

A randomized hash function that uses all the SHA-256 bits solves this
problem: the second run has a non-zero probability of removing k', as
long as the Bloom filter is not completely full.
2020-07-11 09:36:09 +02:00
Jakob Borg
bbda58a29f lib/db: Add test for GC run 2020-07-11 09:22:15 +02:00
Simon Frei
d7fc7008af
lib/protocol: Add BlocksHash to FileInfo string (#6802) 2020-06-27 07:44:33 +02:00
Simon Frei
4881a6336f
lib/db: Correct need accounting on reset (fixes #6784) (#6785) 2020-06-23 07:29:27 +02:00
Simon Frei
8cf9d91ed4
lib: Print nicely rounded durations (#6756) 2020-06-18 10:55:41 +02:00
Simon Frei
cbe0d2fffc
lib/db: Improve error message on meta inconsistency (#6751) 2020-06-17 10:03:39 +02:00
Jakob Borg
a218a69530
lib/db: Use explicit byte type for type prefixes (#6754)
This makes Go 1.15 test/vet happy, avoiding "conversion from untyped int
to string yields a string of one rune" warning where we do
string(KeyTypeWhatever) in namespaced.go.

It also clarifies and enforces the currently allowed range of these
numbers so I think it's fine.
2020-06-17 09:37:07 +02:00
greatroar
df83b84aa1
all: Make all error implementations pointer types (#6726)
This matches the convention of the stdlib and avoids ambiguity: when
customErr{} and &customErr{} both implement error, client code needs to
check for both.

Memory use should remain the same, since storing a non-pointer type in
an interface value still copies the value to the heap.
2020-06-16 09:27:34 +02:00
greatroar
b6a84ecb0c
lib/db: Remove unused keyer methods (#6723)
Co-authored-by: greatroar <@>
2020-06-08 19:19:18 +01:00
André Colomb
46536509d7
lib/protocol: Avoid panic in DeviceIDFromBytes (#6714) 2020-06-07 10:31:12 +02:00
Simon Frei
fac4dec840
lib/db: New VersionList migration fixes (ref #6638) (#6705) 2020-06-02 23:05:41 +02:00
Simon Frei
1f8e6c55f6
lib/db: Refactor to use global list by version (fixes #6372) (#6638)
Group the global list of files by version, instead of having one flat list for all devices. This removes lots of duplicate protocol.Vectors.

Co-authored-by: Jakob Borg <jakob@kastelo.net>
2020-05-30 09:50:23 +02:00
Jakob Borg
94beed5c10
lib/db: Add Badger backend (fixes #5910) (#6250) 2020-05-29 13:43:02 +02:00
Jakob Borg
f78133b8e9
lib/db: Adjust transaction flush sizes downwards (#6686)
This reduces the size of our write batches before we flush them. This
has two effects: reducing the amount of data lost if we crash when
updating the database, and reducing the amount of memory used when we do
large updates without checkpoint (e.g., deleting a folder).

I ran our SyncManyFiles benchmark as it is the one doing most
transactions, however there was no relevant change in any metric (it's
limited by our fsync I expect). This is good as any visible change would
just be a decrease in performance.

I don't have a benchmark on deleting a large folder, taking that part on
trust for now...
2020-05-27 12:15:00 +02:00
Simon Frei
8f344d0915
lib/db: Initialize need meta on first dev occurrence (fixes #6668) (#6669) 2020-05-20 11:01:27 +02:00
Simon Frei
5b34c31cb3
lib/db: Don't panic on seq. coruption when debugging (#6662) 2020-05-18 10:42:51 +02:00
Audrius Butkevicius
a1c5b44c74
lib/model: Fix rename handling (ref #6650) (#6652) 2020-05-16 14:39:27 +02:00
Simon Frei
de9489585f
lib/db: Filter repeat files in one update (ref #6650) (#6651) 2020-05-16 14:34:53 +02:00
Simon Frei
974551375e
lib/db: Dont add symlinks to blocks map (fixes #6637) (#6639) 2020-05-13 20:38:21 +02:00
Jakob Borg
531ceb2b0f
Add indirection for large version vectors. (#6376)
This adds indirection of large version vectors in the same manner as we
already to block lists. The effect is the same: less duplicated data in
some situations.

To mitigate the impact for when this indirection
wouldn't be needed I've added an indirection cutoff for both blocks and
the new version vector stuff: we don't do the indirection at all for
small block lists or small version vectors, instead storing it directly
like we used to do. This is faster for small files and small setups.
2020-05-13 14:28:42 +02:00
Simon Frei
ba4462a70b
lib/db: Fix updating/removing from global (ref #6413) (#6635) 2020-05-13 12:56:49 +02:00
Audrius Butkevicius
decb967969
all: Reorder sequences for better rename detection (#6574) 2020-05-11 20:15:11 +02:00
Simon Frei
a94951becd
lib/db, lib/model: Keep need stats in metadata (ref #5899) (#6413) 2020-05-11 15:07:06 +02:00
Audrius Butkevicius
da99203dcd
lib/db: Fix non-truncated fileinfo loading (#6621) 2020-05-08 20:31:43 +01:00
Jakob Borg
c0c18a568c
lib/db: Hold the bloom filter the right way (fixes #6614) (#6617) 2020-05-08 14:18:00 +02:00
Simon Frei
22242d51be
lib/db: Refactor getting global lists (#6529)
* lib/db: Refactor getting global lists

* VL -> Versions

* keyBuf

* don't touch db migration
2020-05-01 08:30:20 +01:00
greatroar
0e5ba3ca05
lib/db: Upgrade to Blobloom v0.1.1 (#6553)
Now faster and Apache-licensed.
2020-04-20 14:23:36 +02:00
greatroar
44b0f0b456
lib/db: Switch to faster blobloom Bloom filter pkg (#6537) 2020-04-20 09:02:33 +02:00
greatroar
82fbcb96f8
lib/db: Remove unused blockFinder global (#6534) 2020-04-14 17:09:59 +02:00
Simon Frei
0ba3abdee4
lib/db: Handle missed error variable in old schema upgrade (#6528) 2020-04-13 22:58:04 +02:00
Simon Frei
ab92f8520c
cmd/syncthing, lib/db: Store upgrade info to throttle queries (fixes #6513) (#6514) 2020-04-13 10:21:07 +02:00
Jakob Borg
0e67c036bb
lib/db: Make database GC a service, stop on Stop() (#6518)
This makes the GC runner a service that will stop fairly quickly when
told to.

As a bonus, STTRACE=app will print the service tree on the way out,
including any errors they've flagged.
2020-04-12 10:26:57 +02:00
Simon Frei
df318ed370
lib/db: Don't get blocklists on drop and missing continue (ref #6457) (#6502) 2020-04-07 13:13:18 +02:00
Simon Frei
32245435e2
lib/model: Handle deleted items on RO for remote removes (fixes #6432) (#6464) 2020-04-02 16:14:25 +02:00
greatroar
2f26a95973
lib/db, lib/model: Code simplifications (#6484)
NamespacedKV.prefixedKey is still small enough to be inlined.
2020-03-31 14:32:24 +02:00
Simon Frei
7f23de4f03
all: Pass db intervals as args not env vars (#6448) 2020-03-24 13:53:20 +01:00
Jakob Borg
c4abe6f815
lib/db: Don't whack blocks when putting truncated file (#6434)
As of the latest database checker we are again putting files without
blocks. I'm not 100% convinced that's a great idea, but we also do it
for ignored files apparently so it looks like we probably should support
it. This adds an escape hatch that must be manually enabled...
2020-03-20 12:07:14 +01:00
Simon Frei
74706bb02b
lib/db, lib/syncthing: Repair db once on upgrade (ref #6425, #6427) (#6429) 2020-03-19 15:58:32 +01:00
Simon Frei
cf11fa4327
lib/db: Fix removeFromGlobal and no filenames in error (fixes #6427) (#6428) 2020-03-19 14:32:22 +01:00
Simon Frei
40580d8b9b
lib/db: Remove emptied global list in checkGlobals (fixes #6425) (#6426) 2020-03-19 14:30:20 +01:00
Simon Frei
00b2340f9a
lib/db: Checkpoint during schema updates (fixes #6422) (#6424) 2020-03-18 20:33:43 +01:00
Simon Frei
cc2a55892f
lib: Repair sequence inconsistencies (#6367) 2020-03-18 17:34:46 +01:00
Simon Frei
1bd4ea0cbb
lib/db: Don't ignore failures unmarshaling version lists (#6411) 2020-03-16 10:09:27 +01:00
Simon Frei
a1cb1d70c4
lib/db: Use need func in withNeed and simplify (#6412) 2020-03-16 08:45:45 +01:00
Simon Frei
16698b12b1
lib/db: Extend set test with second remote (#6402) 2020-03-11 08:15:45 +01:00
Jakob Borg
2faa1ad360
lib/db: Be more lenient during migration (fixes #6397) (#6398) 2020-03-06 20:50:55 +01:00
Jakob Borg
dd92b2b8f4
all: Tweak error creation (#6391)
- In the few places where we wrap errors, use the new Go 1.13 "%w"
  construction instead of %s or %v.

- Where we create errors with constant strings, consistently use
  errors.New and not fmt.Errorf.

- Remove capitalization from errors in the few places where we had that.
2020-03-03 22:40:00 +01:00
Jakob Borg
c08e253e7c
lib/db: Prevent GC concurrently with migration (fixes #6389) (#6390) 2020-02-29 19:51:32 +01:00
Jakob Borg
5de6f6d349
lib/db: Correct metadata recalculation (fixes #6381) (#6382)
If we decide to recalculate the metadata we shouldn't start from
whatever we loaded from the database, as that data is wrong. We should
start from a clean slate.
2020-02-28 11:16:33 +01:00
Jakob Borg
883497966e lib/db: Remove reference to env var that never existed 2020-02-27 11:21:35 +01:00