Commit Graph

176 Commits

Author SHA1 Message Date
kylosus
302b352d78
lib/fs: Add invalid UTF-8 guards to watcher (fixes #9369) (#9372)
Add invalid UTF-8 guards to fix #9369. Probably not a permanent fix, but
putting it up here in case someone else encounters the same panic.
2024-01-28 19:50:26 +01:00
Jakob Borg
3297624037
lib/ignore: Optimise ignoring directories for filesystem watcher (fixes #9339) (#9340)
This improves the ignore handling so that directories can be fully
ignored (skipped in the watcher) in more cases. Specifically, where the
previous rule was that any complex `!`-pattern would disable skipping
directories, the new rule is that only matches on patterns *after* such
a `!`-pattern disable skipping. That is, the following now does the
intuitive thing:

```
/foo
/bar
!whatever
*
```

- `/foo/**` and `/bar/**` are completely skipped, since there is no
chance anything underneath them could ever be not-ignored
- `!whatever` toggles the "can't skip directories any more" flag
- Anything that matches `*` can't skip directories, because it's
possible we can have `whatever` match something deeper.

To enable this, some refactoring was necessary:

- The "can skip dirs" flag is now a property of the match result, not of
the pattern set as a whole.
- That meant returning a boolean is not good enough, we need to actually
return the entire `Result` (or, like, two booleans but that seemed
uglier and more annoying to use)
- `ShouldIgnore(string) boolean` went away with
`Match(string).IsIgnored()` being the obvious replacement (API
simplification!)
- The watcher then needed to import the `ignore` package (for the
`Result` type), but `fs` imports the watcher and `ignore` imports `fs`.
That's a cycle, so I broke out `Result` into a package of its own so
that it can be safely imported everywhere in things like `type Matcher
interface { Match(string) result.Result }`. There's a fair amount of
stuttering in `result.Result` and maybe we should go with something like
`ignoreresult.R` or so, leaving this open for discussion.

Tests refactored to suit, I think this change is in fact quite well
covered by the existing ones...

Also some noise because a few of the changed files were quite old and
got the `gofumpt` treatment by my editor. Sorry not sorry.

---------

Co-authored-by: Simon Frei <freisim93@gmail.com>
2024-01-15 10:13:22 +00:00
Jakob Borg
e041877488
lib/ignore: Refactor out result type (#9343) 2024-01-13 18:58:23 +01:00
Jakob Borg
c1ec9a8826
lib/fs: Reduce memory usage in xattrs handling (#9251)
This reduces allocations, in number and in size, while getting extended
attributes. This is mostly noticable when there is a large number of new
files to scan and we're running with the default scanProgressInterval --
then a queue of files is built in-memory, and this queue includes
extended attributes as part of file metadata. (Arguable it shouldn't,
but that's a more difficult and involved change.)

With 1M files to scan, each with one extended attribute, current peak
memory usage looks like this:

	Showing nodes accounting for 1425.30MB, 98.19% of 1451.64MB total
	Dropped 1435 nodes (cum <= 7.26MB)
	Showing top 10 nodes out of 54
	      flat  flat%   sum%        cum   cum%
976.56MB 67.27% 67.27% 976.56MB 67.27%
github.com/syncthing/syncthing/lib/fs.getXattr
305.44MB 21.04% 88.31% 305.44MB 21.04%
github.com/syncthing/syncthing/lib/scanner.(*walker).walk.func1
45.78MB 3.15% 91.47% 1045.23MB 72.00%
github.com/syncthing/syncthing/lib/fs.(*BasicFilesystem).GetXattr
22.89MB 1.58% 93.04% 22.89MB 1.58%
github.com/syncthing/syncthing/lib/fs.listXattr
22.89MB 1.58% 94.62% 22.89MB 1.58%
github.com/syncthing/syncthing/lib/protocol.(*PlatformData).SetXattrs
16MB 1.10% 95.72% 16.01MB 1.10%
github.com/syndtr/goleveldb/leveldb/memdb.New

After the change, it's this:

	Showing nodes accounting for 502.32MB, 95.70% of 524.88MB total
	Dropped 1400 nodes (cum <= 2.62MB)
	Showing top 10 nodes out of 91
	      flat  flat%   sum%        cum   cum%
305.43MB 58.19% 58.19% 305.43MB 58.19%
github.com/syncthing/syncthing/lib/scanner.(*walker).walk.func1
45.79MB 8.72% 66.91% 68.68MB 13.09%
github.com/syncthing/syncthing/lib/fs.(*BasicFilesystem).GetXattr
32MB 6.10% 73.01% 32.01MB 6.10%
github.com/syndtr/goleveldb/leveldb/memdb.New
22.89MB 4.36% 77.37% 22.89MB 4.36%
github.com/syncthing/syncthing/lib/fs.listXattr
22.89MB 4.36% 81.73% 22.89MB 4.36%
github.com/syncthing/syncthing/lib/protocol.(*PlatformData).SetXattrs
15.35MB 2.92% 84.66% 15.36MB 2.93%
github.com/syndtr/goleveldb/leveldb/util.(*BufferPool).Get
	   15.28MB  2.91% 87.57%    15.28MB  2.91%  strings.(*Builder).grow

(The usage for xattrs is reduced from 976 MB to 68 MB)
2023-12-04 12:48:17 +01:00
Jakob Borg
4cba99fcd4 lib/fs: Better equality comparison in mtimefs 2023-12-03 16:01:46 +01:00
tomasz1986
16ae1fbe5e
lib/fs: Ignore inode change time on Android (#9177)
lib/fs: Fix conflicts on Android due to fluctuating inode change time

[1] added inode change time to file info in order to support syncing
extended attributes. However, in the case of Android, this inode change
time fluctuates, leading to unexpected conflicts even when the user has
not even touched the files on the Android device itself. Thus, in order
to prevent those conflicts from happening, do not write inode change
time on Android.

[1] 6cac308bcd

Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
2023-10-21 08:24:29 +02:00
Eric P
9553365d31
lib/fs: Properly handle Windows deduplicated files (fixes #9120) (#9168)
### Purpose

Deduplicated files are apparently considered 'irregular' under the hood,
this causes them to simply be ignored by Syncthing. This change is more
of a workaround than a proper fix, as the fix should probably happen in
the underlying libraries? - which may take some time. In the meanwhile,
this change should make deduplicated files be treated as regular files
and be indexed and synced as they should.

### Testing

Create some volume where deduplication is turned on (see the relevant
issue for details, including a proper description of how to reproduce
it). Prior to this change, the deduplicated files were simply ignored
(even by the indexer). After this change, the deduplicated files are
being index and synced properly.
2023-10-11 14:40:55 +02:00
Jakob Borg
051cbdc713
lib/fs, lib/model: Be careful about potentially negative durations (fixes #9112) (#9113)
I don't really understand under what circumstances, but sometimes these
calls panic with a "panic: counter cannot decrease in value" because the
value passed to Add() was negative.
2023-09-20 09:04:47 +02:00
André Colomb
a8cacdca94
lib/versioner: Minor fixes in comments and error message (#9031)
* lib/versioner: Factor out DefaultPath constant.

Replace several instances where .stversions is named literally to all
use the same definition in the versioner package.  Exceptions are the
packages where a cyclic dependency on versioner is impossible, or some
tests which combine the versions base path with other components.

* lib/versioner: Fix comment about trash can in simple versioner.

* lib/versioner: Fix wrong versioning type string in error message.

The error message shows the folder type instead of the versioning
type, although the correct field is used in the comparison.
2023-08-09 07:10:06 +00:00
Jakob Borg
b9c08d3814
all: Add Prometheus-style metrics to expose some internal performance counters (fixes #5175) (#9003) 2023-08-04 19:57:30 +02:00
Jakob Borg
c44de2cd58
lib/fs: Clarify errors for Windows filenames (fixes #8968) (#8969)
With this change, error messages include the offending characters or
name parts. Examples:

    nul.txt: name is invalid, contains Windows reserved name: "nul"
    foo>bar.txt: name is invalid, contains Windows reserved character: ">"
    foo \bar.txt: name is invalid, must not end in space or period on Windows
2023-07-07 11:00:40 +00:00
Jakob Borg
ae176ea9cd
build: Tests should run with Go 1.20 on Windows (#8924)
Tests should run with Go 1.20 on Windows
2023-06-05 10:19:47 +02:00
Ross Smith II
3adfe2f91f lib/fs: Fix root path handling for Windows (fixes #8778)
Co-authored-by: Jakob Borg <jakob@kastelo.net>
2023-05-09 10:01:57 +00:00
Jakob Borg
1103a27337 all: Grand test refactor (fixes #8779, fixes #8799)
This fixes various test issues with Go 1.20.

- Most tests rewritten to use fakefs where possible
- Some tests that were already skipped, or dubious (invasive,
  unmaintainable, unclear what they even tested) have been removed
- Some actual code rewritten to better support testing in fakefs

Co-authored-by: Eric P <eric@kastelo.net>
2023-05-09 10:01:57 +00:00
Alexander Seiler
ddce692f72
all: Correct various typos (#8870) 2023-05-09 08:54:02 +02:00
Dimitri Papadopoulos Orfanos
526e21ae26
all: Fix typos found by codespell (#8833) 2023-03-21 08:07:28 +01:00
Simon Frei
634a3d0e3b
lib/fs: Use io/fs errors as recommended in std lib (#8726) 2022-12-21 23:42:22 +01:00
Jakob Borg
ad0044fec8
lib/fs: Watching is unsupported on android/amd64 (fixes #8709) (#8710) 2022-12-21 22:01:00 +01:00
greatroar
663106ef6e
lib/fs: Optimize WindowsInvalidFilename (#8687)
Replaced strings.Split with the new strings.Cut, which doesn't allocate.

name                           old time/op    new time/op    delta
WindowsInvalidFilenameValid-8     154ns ± 2%      89ns ± 0%   -42.09%  (p=0.000 n=10+9)
WindowsInvalidFilenameNUL-8       124ns ± 2%     124ns ± 1%      ~     (p=0.371 n=8+10)

name                           old alloc/op   new alloc/op   delta
WindowsInvalidFilenameValid-8     16.0B ± 0%      0.0B       -100.00%  (p=0.000 n=10+10)
WindowsInvalidFilenameNUL-8       19.0B ± 0%      3.0B ± 0%   -84.21%  (p=0.000 n=10+10)

name                           old allocs/op  new allocs/op  delta
WindowsInvalidFilenameValid-8      1.00 ± 0%      0.00       -100.00%  (p=0.000 n=10+10)
WindowsInvalidFilenameNUL-8        2.00 ± 0%      1.00 ± 0%   -50.00%  (p=0.000 n=10+10)
2022-11-21 12:58:00 +01:00
Jakob Borg
2a8362d7af
build: Update all dependencies (fixes #8679) (#8680)
Minor API change for the LRU cache in casefs
2022-11-17 21:26:10 +01:00
Jakob Borg
a29605750d
lib/fs: Try to remove read only Windows files (fixes #3744) (#8650)
This happens when folders contain a custom icon.

Co-authored-by: Alexandre Alves <alexandrealvesdb.contact@gmail.com>
2022-11-07 21:33:17 +01:00
Jakob Borg
bf1e418e4a
lib/fs: Let xattr test avoid non-test attributes (fixes #8601) (#8628)
SELinux for example adds security.* attributes by default that we are
not allowed to touch, which causes the test to fail.
2022-11-03 11:57:30 +01:00
Simon Frei
0935886045
lib/fs: Skip xattrs test on EOPNOTSUPP (fixes #8564) (#8565) 2022-10-01 08:48:00 +02:00
Jakob Borg
6cac308bcd
all: Support syncing extended attributes (fixes #2698) (#8513)
This adds support for syncing extended attributes on supported
filesystem on Linux, macOS, FreeBSD and NetBSD. Windows is currently
excluded because the APIs seem onerous and annoying and frankly the uses
cases seem few and far between. On Unixes this also covers ACLs as those
are stored as extended attributes.

Similar to ownership syncing this will optional & opt-in, which two
settings controlling the main behavior: one to "sync" xattrs (read &
write) and another one to "scan" xattrs (only read them so other devices
can "sync" them, but not apply any locally).

Co-authored-by: Tomasz Wilczyński <twilczynski@naver.com>
2022-09-14 09:50:55 +02:00
luzpaz
837ffcfab5
all: Fix various user-facing and non-user-facing typos (#8509)
Found via `codespell -q 3 -S lang,./gui/default/vendor -L benchs,bu,inflight,ro`
2022-08-23 15:44:11 +02:00
Jakob Borg
eb81f7400c
lib/fs: Cache user lookups (#8496) 2022-08-12 07:48:00 +02:00
Jakob Borg
a3c724f2c3
all: Add build constants for runtime.GOOS comparisons (#8442)
all: Add package runtimeos for runtime.GOOS comparisons

I grew tired of hand written string comparisons. This adds generated
constants for the GOOS values, and predefined Is$OS constants that can
be iffed on. In a couple of places I rewrote trivial switch:es to if:s,
and added Illumos where we checked for Solaris (because they are
effectively the same, and if we're going to target one of them that
would be Illumos...).
2022-07-28 19:36:39 +02:00
Jakob Borg
212258d213 lib/fs: Consolidate append in test 2022-07-28 19:05:24 +02:00
deepsource-autofix[bot]
81d8fa1cb5
all: Fix unused method receiver (further) (#8466)
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>
2022-07-28 17:55:29 +02:00
deepsource-autofix[bot]
755d21953f
all: Remove unused method receivers (#8462)
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>
2022-07-28 17:32:45 +02:00
deepsource-autofix[bot]
5130c414da
all: Unused parameter should be replaced by underscore (#8464)
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>
2022-07-28 17:17:29 +02:00
deepsource-autofix[bot]
80ec4acb53
all: Fix check for empty string (#8456)
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>
2022-07-28 16:51:03 +02:00
Jakob Borg
adce6fa473
all: Support syncing ownership (fixes #1329) (#8434)
This adds support for syncing ownership on Unixes and on Windows. The
scanner always picks up ownership information, but it is not applied
unless the new folder option "Sync Ownership" is set.

Ownership data is stored in a new FileInfo field called "platform data". This
is intended to hold further platform-specific data in the future
(specifically, extended attributes), which is why the whole design is a
bit overkill for just ownership.
2022-07-26 08:24:58 +02:00
Eng Zer Jun
bc27aa12cd
all: use T.TempDir to create temporary test directory (#8280)
This commit replaces `os.MkdirTemp` with `t.TempDir` in tests. The
directory created by `t.TempDir` is automatically removed when the test
and all its subtests complete.

Prior to this commit, temporary directory created using `os.MkdirTemp`
needs to be removed manually by calling `os.RemoveAll`, which is omitted
in some tests. The error handling boilerplate e.g.
	defer func() {
		if err := os.RemoveAll(dir); err != nil {
			t.Fatal(err)
		}
	}
is also tedious, but `t.TempDir` handles this for us nicely.

Reference: https://pkg.go.dev/testing#T.TempDir
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2022-04-15 07:44:06 +04:00
Simon Frei
db72579f0e
lib: Get rid of buggy filesystem wrapping (#8257) 2022-04-10 20:55:05 +02:00
Simon Frei
edc3a77b98
lib/fs, lib/model: Add warning about kqueue resource usage (fixes #7855) (#8249) 2022-04-05 21:32:06 +02:00
Greg
53926a1ae6
lib/fs: Do not follow symlinks in watcher on solaris (fixes #8020) (#8223) 2022-03-24 08:36:43 +01:00
Jonathan
d37a5b03f1
lib/fs: Handle permission change events on macos (fixes #7924) (#8150) 2022-01-30 17:21:21 +01:00
Simon Frei
8763fb05ec
lib/fs: Improve error messages checking file paths (fixes #7942) (#8148) 2022-01-29 18:47:55 +01:00
greatroar
8a1e54d58a lib/fs: Optimize Canonicalize
When pathSep is a constant, the compiler precomputes pathSep+pathSep and
".."+pathSep instead of emitting function calls to compute "//" and
"../". Benchmark results in lib/osutil:

name                old time/op    new time/op    delta
TraversesSymlink-8    8.86µs ± 3%    8.53µs ± 4%  -3.79%  (p=0.000 n=18+20)

name                old alloc/op   new alloc/op   delta
TraversesSymlink-8    1.06kB ± 0%    1.06kB ± 0%    ~     (all equal)

name                old allocs/op  new allocs/op  delta
TraversesSymlink-8      15.0 ± 0%      15.0 ± 0%    ~     (all equal)
2021-11-23 21:25:29 +01:00
greatroar
3e032c4da6 lib/fs: optimize Windows path checking/sanitizing
name                          old time/op    new time/op    delta
WindowsInvalidFilenameValid-8     875ns ± 1%     150ns ± 1%  -82.84%  (p=0.000 n=9+9)
WindowsInvalidFilenameNUL-8       276ns ± 4%     121ns ± 3%  -56.26%  (p=0.000 n=10+10)

name                          old alloc/op   new alloc/op   delta
WindowsInvalidFilenameValid-8     32.0B ± 0%     16.0B ± 0%  -50.00%  (p=0.000 n=10+10)
WindowsInvalidFilenameNUL-8       32.0B ± 0%     19.0B ± 0%  -40.62%  (p=0.000 n=10+10)

name                          old allocs/op  new allocs/op  delta
WindowsInvalidFilenameValid-8      2.00 ± 0%      1.00 ± 0%  -50.00%  (p=0.000 n=10+10)
WindowsInvalidFilenameNUL-8        2.00 ± 0%      2.00 ± 0%     ~     (all equal)
2021-11-23 21:25:29 +01:00
Jakob Borg
4b750b6dc3
all: Remove usage of deprecated io/ioutil (#7971)
As of Go 1.16 io/ioutil is deprecated. This replaces usage with the
corresponding functions in package os and package io.
2021-11-22 08:59:47 +01:00
Jakob Borg
f1bf4d899a lib/model: Correct handling of fakefs cache
We looked under one cache key, then stored under another...
2021-11-17 12:52:10 +01:00
greatroar
ed98039aa5
lib/fs: Optimize TempName + some cosmetic changes (#7911) 2021-08-29 10:47:53 +02:00
Jakob Borg
d47745a86b
all: Update build constraints to Go 1.17 style (#7894) 2021-08-17 10:10:41 +02:00
tomasz1986
5a1f6cb813
lib/fs: Improve case conflict error message (fixes #7827) (#7829) 2021-08-01 22:44:49 +02:00
Simon Frei
08e3cd1cce
lib/fs: Set expiry after DirNames in case-fs (#7794) 2021-06-27 08:30:02 +02:00
Anur
e7f8538e4d
lib/fs: Add bitmasks for Darwin to handle change to empty files (fixes #7731) (#7756) 2021-06-09 12:57:06 +02:00
Simon Frei
df48276300
lib/model: Ensure indexes are only received after checking IDs (ref #7649) (#7689) 2021-06-03 14:58:50 +02:00
greatroar
0e52ce830a
lib/fs: Fix UnicodeLowercaseNormalized on lowercase NFD (#7692)
Co-authored-by: greatroar <@>
2021-05-17 20:43:07 +02:00