Commit Graph

9 Commits

Author SHA1 Message Date
Simon Frei
9be6f1a70e lib/fs: Consider options in case-fs caching (fixes #7371) (#7381) 2021-02-22 18:27:40 +01:00
Simon Frei
e19d6e993d
lib/fs: Cache all real-case results (fixes #7270) (#7286) 2021-01-27 19:25:34 +01:00
Jakob Borg
36acaf5e21
lib/fs: Avoid blocking new caseFs creation while waiting to drop cache (fixes #7273) (#7275) 2021-01-13 17:45:29 +01:00
Jakob Borg
8f199e12b3
lib/fs, lib/model: Reduce lock contention on NewCaseFilesystem (fixes #7268) (#7269) 2021-01-12 16:22:21 +01:00
Simon Frei
a744dee94c
lib/fs: Correct wrapping order for meaningful log-caller (#7209) 2020-12-21 13:01:34 +01:00
Jakob Borg
780fb3bac1
lib/fs: More efficient casefs cache (#6974)
This changes the cache to cache less things, yet retain the required
efficiency for our walk usecase. This uses less memory.

Specifically, instead of keeping result and child caches for each path
level, only keep a single cached child. In practice our operations are
depth-first, or almost depth-first, and then we retain the same hit
ratio for a smaller cache size.

I improved the benchmark so that it counts the Lstat and DirNames
operations performed, and they do not change significantly. The amount
of allocated memory is reduced by 20% and the walk itself is actually
slightly faster.

This also removes the clear based on number of cached names (as that is
not a thing any more) and the timer based clear (which was unused). This
means we'll retain the last cache state forever until it's cleared by a
write operation, but we did that before too and that state is now a lot
smaller...

The overhead compared to not using a casefs, for our typical "double
walk" (walk the tree then stat everything again) is 2x the dirnames we
would otherwise call, and no overhead on the stats (unchanged from old
implementation)

```
name                         old time/op         new time/op         delta
WalkCaseFakeFS100k/rawfs-8           306ms ± 1%          305ms ± 2%     ~     (p=0.182 n=9+10)
WalkCaseFakeFS100k/casefs-8          579ms ± 5%          557ms ± 1%   -3.77%  (p=0.000 n=10+10)

name                         old B/entry         new B/entry         delta
WalkCaseFakeFS100k/rawfs-8             590 ± 0%            590 ± 0%     ~     (all equal)
WalkCaseFakeFS100k/casefs-8          1.09k ± 0%          0.87k ± 0%  -19.98%  (p=0.000 n=10+10)

name                         old DirNames/entry  new DirNames/entry  delta
WalkCaseFakeFS100k/rawfs-8            0.51 ± 0%           0.51 ± 0%     ~     (all equal)
WalkCaseFakeFS100k/casefs-8           1.02 ± 0%           1.02 ± 0%     ~     (all equal)

name                         old DirNames/op     new DirNames/op     delta
WalkCaseFakeFS100k/rawfs-8           51.2k ± 0%          51.2k ± 0%     ~     (all equal)
WalkCaseFakeFS100k/casefs-8           102k ± 0%           102k ± 0%     ~     (all equal)

name                         old Lstat/entry     new Lstat/entry     delta
WalkCaseFakeFS100k/rawfs-8            3.02 ± 0%           3.02 ± 0%     ~     (all equal)
WalkCaseFakeFS100k/casefs-8           3.02 ± 0%           3.02 ± 0%     ~     (all equal)

name                         old Lstat/op        new Lstat/op        delta
WalkCaseFakeFS100k/rawfs-8            302k ± 0%           302k ± 0%     ~     (all equal)
WalkCaseFakeFS100k/casefs-8           302k ± 0%           302k ± 0%     ~     (all equal)

name                         old allocs/entry    new allocs/entry    delta
WalkCaseFakeFS100k/rawfs-8            15.7 ± 0%           15.7 ± 0%     ~     (all equal)
WalkCaseFakeFS100k/casefs-8           27.5 ± 0%           26.1 ± 0%   -5.09%  (p=0.000 n=10+10)

name                         old ns/entry        new ns/entry        delta
WalkCaseFakeFS100k/rawfs-8           2.02k ± 1%          2.02k ± 2%     ~     (p=0.163 n=9+10)
WalkCaseFakeFS100k/casefs-8          3.83k ± 5%          3.68k ± 1%   -3.77%  (p=0.000 n=10+10)

name                         old alloc/op        new alloc/op        delta
WalkCaseFakeFS100k/rawfs-8          89.2MB ± 0%         89.2MB ± 0%     ~     (p=0.364 n=9+10)
WalkCaseFakeFS100k/casefs-8          164MB ± 0%          131MB ± 0%  -19.97%  (p=0.000 n=10+10)

name                         old allocs/op       new allocs/op       delta
WalkCaseFakeFS100k/rawfs-8           2.38M ± 0%          2.38M ± 0%     ~     (all equal)
WalkCaseFakeFS100k/casefs-8          4.16M ± 0%          3.95M ± 0%   -5.05%  (p=0.000 n=10+10)
```
2020-09-09 14:38:39 +02:00
Simon Frei
e3cd9219b8
lib/model: Don't fail over case-conflict on tempfile (fixes #6973) (#6975) 2020-09-09 11:47:14 +02:00
Simon Frei
b628ec5054
lib/fs: Use generic case resolver on windows (fixes #6968) (#6969) 2020-09-08 13:10:26 +02:00
Simon Frei
932d8c69de lib/fs: Properly handle case insensitive systems (fixes #1787, fixes #2739, fixes #5708)
With this change we emulate a case sensitive filesystem on top of
insensitive filesystems. This means we correctly pick up case-only renames
and throw a case conflict error when there would be multiple files differing
only in case.

This safety check has a small performance hit (about 20% more filesystem
operations when scanning for changes). The new advanced folder option
`caseSensitiveFS` can be used to disable the safety checks, retaining the
previous behavior on systems known to be fully case sensitive.

Co-authored-by: Jakob Borg <jakob@kastelo.net>
2020-07-28 11:15:11 +02:00