Commit Graph

323 Commits

Author SHA1 Message Date
Jakob Borg
42e677c055
lib/model, lib/protocol: Index sending/receiving debugging (#9657)
This adds guardrails to the index sending and receiving, to verify that
what we thinks is happening is what actually happens.
2024-08-28 15:00:19 +02:00
Gusted
356c5055ad
lib/sha256: Remove it (#9643)
### Purpose

Remove the `lib/sha256` package, because it's no longer necessary. Go's
standard library now has the same performance and is on par with
`sha256-simd` since [Since Go
1.21](1a64574f42).
Therefore using `sha256-simd` has no benefits anymore.

ARM already has optimized sha256 assembly code since
7b8a7f8272,
`sha256-simd` published their results before that optimized assembly was
implemented,
f941fedda8.
The assembly looks very similar and the benchmarks in the Go commit
match that of `sha256-simd`.

This patch removes all of the related code of `lib/sha256` and makes
`crypto/sha256` the 'default'.

Benchmark of `sha256-simd` and `crypto/sha256`:
<details>

```
cpu: AMD Ryzen 5 3600X 6-Core Processor
                │  simd.txt   │               go.txt                │
                │   sec/op    │    sec/op     vs base               │
Hash/8Bytes-12    63.25n ± 1%    73.38n ± 1%  +16.02% (p=0.002 n=6)
Hash/64Bytes-12   98.73n ± 1%   105.30n ± 1%   +6.65% (p=0.002 n=6)
Hash/1K-12        567.2n ± 1%    572.8n ± 1%   +0.99% (p=0.002 n=6)
Hash/8K-12        4.062µ ± 1%    4.062µ ± 1%        ~ (p=0.396 n=6)
Hash/1M-12        512.1µ ± 0%    510.6µ ± 1%        ~ (p=0.485 n=6)
Hash/5M-12        2.556m ± 1%    2.564m ± 0%        ~ (p=0.093 n=6)
Hash/10M-12       5.112m ± 0%    5.127m ± 0%        ~ (p=0.093 n=6)
geomean           13.82µ         14.27µ        +3.28%

                │   simd.txt   │               go.txt                │
                │     B/s      │     B/s       vs base               │
Hash/8Bytes-12    120.6Mi ± 1%   104.0Mi ± 1%  -13.81% (p=0.002 n=6)
Hash/64Bytes-12   618.2Mi ± 1%   579.8Mi ± 1%   -6.22% (p=0.002 n=6)
Hash/1K-12        1.682Gi ± 1%   1.665Gi ± 1%   -0.98% (p=0.002 n=6)
Hash/8K-12        1.878Gi ± 1%   1.878Gi ± 1%        ~ (p=0.310 n=6)
Hash/1M-12        1.907Gi ± 0%   1.913Gi ± 1%        ~ (p=0.485 n=6)
Hash/5M-12        1.911Gi ± 1%   1.904Gi ± 0%        ~ (p=0.093 n=6)
Hash/10M-12       1.910Gi ± 0%   1.905Gi ± 0%        ~ (p=0.093 n=6)
geomean           1.066Gi        1.032Gi        -3.18%
```

</details>


### Testing

Compiled and tested on Linux.

### Documentation

https://github.com/syncthing/docs/pull/874
2024-08-10 12:58:20 +01:00
luchenhan
57d399317e
lib/db: Correct function name in comments (#9520) 2024-05-16 07:02:57 +00:00
Jakob Borg
61b94b9ea5
lib/db: Drop indexes for outgoing data to force refresh (ref #9496) (#9502)
### Purpose

Resend our indexes since we fixed that index-sending issue.

I made a new thing to only drop the non-local-device index IDs, i.e.,
those for other devices. This means we will see a mismatch and resend
all indexes, but they will not. This is somewhat cleaner as it avoids
resending everything twice when two devices are upgraded, and in any
case, we have no reason to force a resend of incoming indexes here.

### Testing

It happens on my computer...
2024-04-08 11:14:27 +02:00
Jakob Borg
2e7c03420f
lib/db: Hold update lock while taking snapshot (#9496) 2024-04-05 21:32:43 +02:00
Jakob Borg
96c30f8387
lib/model, lib/protocol: Remove FileInfoBatch reuse behavior (#9399) 2024-02-10 19:16:27 +01:00
Jakob Borg
f16817632f
lib/api: Improve folder summary event, verbose service (#9370)
This makes a couple of small improvements to the folder summary
mechanism:

- The folder summary includes the local and remote sequence numbers in
clear text, rather than some odd sum that I'm not sure what it was
intended to represent.
- The folder summary event is generated when appropriate, regardless of
whether there is an event listener. We did this before because
generating it was expensive, and we wanted to avoid doing it
unnecessarily. Nowadays, however, it's mostly just reading out
pre-calculated metadata, and anyway, it's nice if it shows up reliably
when running with -verbose.

The point of all this is to make it easier to use these events to judge
when devices are, in fact, in sync. As-is, if I'm looking at two
devices, it's very difficult to reliably determine if they are in sync
or not. The reason is that while we can ask device A if it thinks it's
in sync, we can't see if the answer is "yes" because it has processed
all changes from B, or if it just doesn't know about the changes from B
yet. With proper sequence numbers in the event we can compare the two
and determine the truth. This makes testing a lot easier.
2024-01-31 08:24:39 +01:00
Jakob Borg
acd767b30b
all: Remove lib/util package (#9049)
Grab-bag packages are nasty, this cleans it up a little by splitting it
into topical packages sempahore, netutil, stringutil, structutil.
2023-08-21 19:44:33 +02:00
guangwu
27aba3567b
all: Minor staticcheck fixes (#8939) 2023-06-19 06:50:53 +00:00
Eng Zer Jun
089320aadc
lib: replace Readdir(-1) with os.ReadDir (#8901) 2023-05-11 15:35:52 +00:00
Jakob Borg
ab8e6a82ab
lib/api: Expose blocksHash in file info (#8810)
This adds the BlocksHash field from the FileInfo to our API output. It
can be useful for debugging, or for external tools. I'm intentionally
leaving it as an opaque base64 string because no meaning should be
derived from it: it's just a string.
2023-03-06 15:37:15 +01:00
Jakob Borg
6cac308bcd
all: Support syncing extended attributes (fixes #2698) (#8513)
This adds support for syncing extended attributes on supported
filesystem on Linux, macOS, FreeBSD and NetBSD. Windows is currently
excluded because the APIs seem onerous and annoying and frankly the uses
cases seem few and far between. On Unixes this also covers ACLs as those
are stored as extended attributes.

Similar to ownership syncing this will optional & opt-in, which two
settings controlling the main behavior: one to "sync" xattrs (read &
write) and another one to "scan" xattrs (only read them so other devices
can "sync" them, but not apply any locally).

Co-authored-by: Tomasz Wilczyński <twilczynski@naver.com>
2022-09-14 09:50:55 +02:00
luzpaz
837ffcfab5
all: Fix various user-facing and non-user-facing typos (#8509)
Found via `codespell -q 3 -S lang,./gui/default/vendor -L benchs,bu,inflight,ro`
2022-08-23 15:44:11 +02:00
Jakob Borg
209e68c1ba
build: Update quic-go for Go 1.19 (#8483)
Also adds idle time and keepalive parameters because how this is
configured has changed in the new package version. The values are those
that seems like might already be default, if keep-alives were enabled,
which is not obvious from the doc comments.

Also, Go 1.19 gofmt reformatting of comments.
2022-08-03 15:43:26 +02:00
deepsource-autofix[bot]
755d21953f
all: Remove unused method receivers (#8462)
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>
2022-07-28 17:32:45 +02:00
deepsource-autofix[bot]
5130c414da
all: Unused parameter should be replaced by underscore (#8464)
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>
2022-07-28 17:17:29 +02:00
Jakob Borg
adce6fa473
all: Support syncing ownership (fixes #1329) (#8434)
This adds support for syncing ownership on Unixes and on Windows. The
scanner always picks up ownership information, but it is not applied
unless the new folder option "Sync Ownership" is set.

Ownership data is stored in a new FileInfo field called "platform data". This
is intended to hold further platform-specific data in the future
(specifically, extended attributes), which is why the whole design is a
bit overkill for just ownership.
2022-07-26 08:24:58 +02:00
greatroar
d00a30069a
lib/db: Constant/unused args and return values, double it.Release (#8259) 2022-04-27 20:32:44 +02:00
Simon Frei
db72579f0e
lib: Get rid of buggy filesystem wrapping (#8257) 2022-04-10 20:55:05 +02:00
greatroar
26eaedc491
lib/db, lib/discover: Minor cleanup (#8217) 2022-03-14 22:48:10 +01:00
greatroar
d0fd6c6c82
lib/db: Make err(Closed|NotFound) values (#8215) 2022-03-13 20:53:34 +01:00
Simon Frei
635085d139
lib/db, lib/model: Remove filesystem state from FileSet (fixes #7850) (#8151) 2022-01-31 10:12:52 +01:00
Simon Frei
591e4d8af1
gui, lib: Fix tracking deleted locally-changed on encrypted (fixes #7715) (#7726) 2021-11-10 09:46:21 +01:00
Jakob Borg
73c5184518
build: Update all dependencies (#7968)
Also add a script step for how to do this, as frankly I forget each time
what the canonical way is.

`go run build.go update-deps`
2021-09-26 12:08:23 +02:00
greatroar
f96c211198
lib/db: Replace SipHash with hash/maphash (#7962) 2021-09-24 21:26:07 +02:00
greatroar
de719ac409
lib/db: Inline sync.Once in releaser (#7938)
Having a pointer to a Once suggests that it is shared with other
objects, but it never is.
2021-09-10 09:58:17 +02:00
greatroar
7fa141ea39
all: Unused args, retvals, assignments (#7926) 2021-09-08 00:11:16 +02:00
Simon Frei
952f3ffb0c
lib/db/backend: Prevent panic in errors.As (#7873) 2021-08-05 11:25:20 +01:00
Simon Frei
69ce121267
lib/db: Missing event-logger in write-transaction (#7793) 2021-06-27 08:43:49 +02:00
Simon Frei
aeca1fb575
lib/db: Check if sequences change when repairing metadata (#7770) 2021-06-17 13:53:39 +02:00
Simon Frei
ac2988a485
gui, lib: Handle pw adding remote encrypted folder (fixes #7705) (#7772) 2021-06-17 13:53:02 +02:00
Simon Frei
23a0e18292
lib/db: Fix accounting bug when dropping indexes (#7774) 2021-06-17 10:15:11 +02:00
Simon Frei
1e7a3997e3
lib/db, lib/model: Improve error handling on pending items (#7754) 2021-06-09 13:35:17 +02:00
Simon Frei
1a22689328
lib/db: Add failure reports to failures iterating over hashes (#7755) 2021-06-07 23:10:35 +02:00
Jakob Borg
ce65aea0ab
lib/db: Use a more concurrent GC (fixes #7722) (#7750)
This changes the GC mechanism so that the first pass (which reads all
FileInfos to populate bloom filters with block & version hashes) can
happen concurrently with normal database operations.

The big gcMut still exists, and we grab it temporarily to block all
other modifications while we set up the bloom filters. We then release
the lock and let other things happen, with those other things also
updating the bloom filters as required. Once the first phase is done we
again grab the gcMut, knowing that we are the sole modifier of the
database, and do the cleanup.

I also removed the final compaction step.
2021-06-07 10:52:06 +02:00
André Colomb
45edad867c
all: Allow dismissing pending devices / folders without ignoring (fixes #7700) (#7712) 2021-06-07 10:29:24 +02:00
greatroar
95c9561e97
lib/db: Clean up Timer and wait for logging before return in GC (#7720) 2021-05-31 09:50:21 +02:00
Simon Frei
58592e3ef1
lib/db: Add logging for GC (#7707) 2021-05-22 21:36:43 +02:00
Simon Frei
5bdb6798a9
all: Regenerate proto (#7696) 2021-05-19 13:30:20 +02:00
Simon Frei
0d054f9b64
lib/model: Don't use empty folder cfg for index sender (fixes #7649) (#7671) 2021-05-15 11:13:39 +02:00
Audrius Butkevicius
87a0eecc31
lib/fs, lib/api, lib/model: Expose mtime remappings as part of /db/file (#7624)
* lib/fs, lib/api, lib/model: Expose mtime remappings as part of /db/file

* Fix wrong error returned by CLI

* Gofmt

* Better names

* Review comments

* Review comments
2021-05-03 11:28:25 +01:00
Simon Frei
dd39556759
lib: Revert needing invalid files (fixes #7608, ref #7476) (#7609) 2021-04-29 22:01:46 +02:00
Jakob Borg
74823e81e9
all: Deprecate TLS 1.2 on sync connections (fixes #7594) (#7598)
This makes us use TLS 1.3+ on sync connections by default. A new option
`insecureAllowOldTLSVersions` exists to allow communication with TLS
1.2-only clients (roughly Syncthing 1.2.2 and older). Even with that
option set you get a slightly simplified setup, with the cipher suite
order fixed instead of auto detected.
2021-04-26 10:04:35 +02:00
Simon Frei
ec0a66c75b
lib/db, lib/model: Refactor removing expired pending folders (#7537) 2021-04-11 15:24:08 +02:00
Simon Frei
f30f9c50f8
lib/db: Handle indirection error repairing sequences (fixes #7026) (#7525) 2021-04-05 10:24:16 +02:00
André Colomb
fb2d85b9d5
lib/db: Fix comparison of pending folder timestamps (fixes #7532) (#7535) 2021-03-30 21:24:53 +02:00
Simon Frei
7f0d4f6ba8
lib/db: Don't panic debugging an inexistent file (#7534) 2021-03-30 20:06:01 +02:00
Jakob Borg
3ac858b150
all: Remove miscellaneous vestigial code (#7495) 2021-03-17 22:23:12 +01:00
Jakob Borg
6e5514419d
lib/db: Fix some omitted error checks, unused variable (#7489) 2021-03-17 21:41:07 +01:00
Simon Frei
273ee09925
lib/db, lib/model: Allow needing invalid files (fixes #7474) (#7476) 2021-03-15 07:58:01 +01:00