mirror of
https://github.com/octoleo/syncthing.git
synced 2024-12-22 19:08:58 +00:00
c1ec9a8826
This reduces allocations, in number and in size, while getting extended attributes. This is mostly noticable when there is a large number of new files to scan and we're running with the default scanProgressInterval -- then a queue of files is built in-memory, and this queue includes extended attributes as part of file metadata. (Arguable it shouldn't, but that's a more difficult and involved change.) With 1M files to scan, each with one extended attribute, current peak memory usage looks like this: Showing nodes accounting for 1425.30MB, 98.19% of 1451.64MB total Dropped 1435 nodes (cum <= 7.26MB) Showing top 10 nodes out of 54 flat flat% sum% cum cum% 976.56MB 67.27% 67.27% 976.56MB 67.27% github.com/syncthing/syncthing/lib/fs.getXattr 305.44MB 21.04% 88.31% 305.44MB 21.04% github.com/syncthing/syncthing/lib/scanner.(*walker).walk.func1 45.78MB 3.15% 91.47% 1045.23MB 72.00% github.com/syncthing/syncthing/lib/fs.(*BasicFilesystem).GetXattr 22.89MB 1.58% 93.04% 22.89MB 1.58% github.com/syncthing/syncthing/lib/fs.listXattr 22.89MB 1.58% 94.62% 22.89MB 1.58% github.com/syncthing/syncthing/lib/protocol.(*PlatformData).SetXattrs 16MB 1.10% 95.72% 16.01MB 1.10% github.com/syndtr/goleveldb/leveldb/memdb.New After the change, it's this: Showing nodes accounting for 502.32MB, 95.70% of 524.88MB total Dropped 1400 nodes (cum <= 2.62MB) Showing top 10 nodes out of 91 flat flat% sum% cum cum% 305.43MB 58.19% 58.19% 305.43MB 58.19% github.com/syncthing/syncthing/lib/scanner.(*walker).walk.func1 45.79MB 8.72% 66.91% 68.68MB 13.09% github.com/syncthing/syncthing/lib/fs.(*BasicFilesystem).GetXattr 32MB 6.10% 73.01% 32.01MB 6.10% github.com/syndtr/goleveldb/leveldb/memdb.New 22.89MB 4.36% 77.37% 22.89MB 4.36% github.com/syncthing/syncthing/lib/fs.listXattr 22.89MB 4.36% 81.73% 22.89MB 4.36% github.com/syncthing/syncthing/lib/protocol.(*PlatformData).SetXattrs 15.35MB 2.92% 84.66% 15.36MB 2.93% github.com/syndtr/goleveldb/leveldb/util.(*BufferPool).Get 15.28MB 2.91% 87.57% 15.28MB 2.91% strings.(*Builder).grow (The usage for xattrs is reduced from 976 MB to 68 MB) |
||
|---|---|---|
| .. | ||
| basicfs_copy_range_copyfilerange.go | ||
| basicfs_copy_range_duplicateextents.go | ||
| basicfs_copy_range_ioctl.go | ||
| basicfs_copy_range_sendfile.go | ||
| basicfs_copy_range.go | ||
| basicfs_fileinfo_bsdish.go | ||
| basicfs_fileinfo_linuxish.go | ||
| basicfs_fileinfo_unix.go | ||
| basicfs_fileinfo_windows.go | ||
| basicfs_lstat_broken.go | ||
| basicfs_lstat_regular.go | ||
| basicfs_lstat_windows.go | ||
| basicfs_platformdata_unix.go | ||
| basicfs_platformdata_windows.go | ||
| basicfs_test.go | ||
| basicfs_unix.go | ||
| basicfs_watch_errors_linux.go | ||
| basicfs_watch_errors_others.go | ||
| basicfs_watch_eventtypes_darwin.go | ||
| basicfs_watch_eventtypes_fen.go | ||
| basicfs_watch_eventtypes_inotify.go | ||
| basicfs_watch_eventtypes_kqueue.go | ||
| basicfs_watch_eventtypes_other.go | ||
| basicfs_watch_eventtypes_readdcw.go | ||
| basicfs_watch_notkqueue.go | ||
| basicfs_watch_test.go | ||
| basicfs_watch_unsupported.go | ||
| basicfs_watch.go | ||
| basicfs_windows_test.go | ||
| basicfs_windows.go | ||
| basicfs_xattr_bsdish.go | ||
| basicfs_xattr_linuxish.go | ||
| basicfs_xattr_unix.go | ||
| basicfs_xattr_unsupported.go | ||
| basicfs.go | ||
| casefs_test.go | ||
| casefs.go | ||
| copyrangemethod.go | ||
| copyrangemethod.pb.go | ||
| debug.go | ||
| errorfs.go | ||
| fakefs_test.go | ||
| fakefs.go | ||
| filesystem_copy_range_allwithfallback.go | ||
| filesystem_copy_range_standard.go | ||
| filesystem_copy_range_test.go | ||
| filesystem_copy_range.go | ||
| filesystem_test.go | ||
| filesystem.go | ||
| folding_test.go | ||
| folding.go | ||
| logfs.go | ||
| metrics.go | ||
| mtimefs_test.go | ||
| mtimefs.go | ||
| platform_common.go | ||
| tempname_test.go | ||
| tempname.go | ||
| types.go | ||
| types.pb.go | ||
| util_test.go | ||
| util.go | ||
| walkfs_test.go | ||
| walkfs.go | ||