mirror of
https://github.com/octoleo/syncthing.git
synced 2025-02-03 12:28:28 +00:00
c1ec9a8826
This reduces allocations, in number and in size, while getting extended attributes. This is mostly noticable when there is a large number of new files to scan and we're running with the default scanProgressInterval -- then a queue of files is built in-memory, and this queue includes extended attributes as part of file metadata. (Arguable it shouldn't, but that's a more difficult and involved change.) With 1M files to scan, each with one extended attribute, current peak memory usage looks like this: Showing nodes accounting for 1425.30MB, 98.19% of 1451.64MB total Dropped 1435 nodes (cum <= 7.26MB) Showing top 10 nodes out of 54 flat flat% sum% cum cum% 976.56MB 67.27% 67.27% 976.56MB 67.27% github.com/syncthing/syncthing/lib/fs.getXattr 305.44MB 21.04% 88.31% 305.44MB 21.04% github.com/syncthing/syncthing/lib/scanner.(*walker).walk.func1 45.78MB 3.15% 91.47% 1045.23MB 72.00% github.com/syncthing/syncthing/lib/fs.(*BasicFilesystem).GetXattr 22.89MB 1.58% 93.04% 22.89MB 1.58% github.com/syncthing/syncthing/lib/fs.listXattr 22.89MB 1.58% 94.62% 22.89MB 1.58% github.com/syncthing/syncthing/lib/protocol.(*PlatformData).SetXattrs 16MB 1.10% 95.72% 16.01MB 1.10% github.com/syndtr/goleveldb/leveldb/memdb.New After the change, it's this: Showing nodes accounting for 502.32MB, 95.70% of 524.88MB total Dropped 1400 nodes (cum <= 2.62MB) Showing top 10 nodes out of 91 flat flat% sum% cum cum% 305.43MB 58.19% 58.19% 305.43MB 58.19% github.com/syncthing/syncthing/lib/scanner.(*walker).walk.func1 45.79MB 8.72% 66.91% 68.68MB 13.09% github.com/syncthing/syncthing/lib/fs.(*BasicFilesystem).GetXattr 32MB 6.10% 73.01% 32.01MB 6.10% github.com/syndtr/goleveldb/leveldb/memdb.New 22.89MB 4.36% 77.37% 22.89MB 4.36% github.com/syncthing/syncthing/lib/fs.listXattr 22.89MB 4.36% 81.73% 22.89MB 4.36% github.com/syncthing/syncthing/lib/protocol.(*PlatformData).SetXattrs 15.35MB 2.92% 84.66% 15.36MB 2.93% github.com/syndtr/goleveldb/leveldb/util.(*BufferPool).Get 15.28MB 2.91% 87.57% 15.28MB 2.91% strings.(*Builder).grow (The usage for xattrs is reduced from 976 MB to 68 MB)