Open Source Continuous File Synchronization
Go to file
Jakob Borg c1ec9a8826
lib/fs: Reduce memory usage in xattrs handling (#9251)
This reduces allocations, in number and in size, while getting extended
attributes. This is mostly noticable when there is a large number of new
files to scan and we're running with the default scanProgressInterval --
then a queue of files is built in-memory, and this queue includes
extended attributes as part of file metadata. (Arguable it shouldn't,
but that's a more difficult and involved change.)

With 1M files to scan, each with one extended attribute, current peak
memory usage looks like this:

	Showing nodes accounting for 1425.30MB, 98.19% of 1451.64MB total
	Dropped 1435 nodes (cum <= 7.26MB)
	Showing top 10 nodes out of 54
	      flat  flat%   sum%        cum   cum%
976.56MB 67.27% 67.27% 976.56MB 67.27%
github.com/syncthing/syncthing/lib/fs.getXattr
305.44MB 21.04% 88.31% 305.44MB 21.04%
github.com/syncthing/syncthing/lib/scanner.(*walker).walk.func1
45.78MB 3.15% 91.47% 1045.23MB 72.00%
github.com/syncthing/syncthing/lib/fs.(*BasicFilesystem).GetXattr
22.89MB 1.58% 93.04% 22.89MB 1.58%
github.com/syncthing/syncthing/lib/fs.listXattr
22.89MB 1.58% 94.62% 22.89MB 1.58%
github.com/syncthing/syncthing/lib/protocol.(*PlatformData).SetXattrs
16MB 1.10% 95.72% 16.01MB 1.10%
github.com/syndtr/goleveldb/leveldb/memdb.New

After the change, it's this:

	Showing nodes accounting for 502.32MB, 95.70% of 524.88MB total
	Dropped 1400 nodes (cum <= 2.62MB)
	Showing top 10 nodes out of 91
	      flat  flat%   sum%        cum   cum%
305.43MB 58.19% 58.19% 305.43MB 58.19%
github.com/syncthing/syncthing/lib/scanner.(*walker).walk.func1
45.79MB 8.72% 66.91% 68.68MB 13.09%
github.com/syncthing/syncthing/lib/fs.(*BasicFilesystem).GetXattr
32MB 6.10% 73.01% 32.01MB 6.10%
github.com/syndtr/goleveldb/leveldb/memdb.New
22.89MB 4.36% 77.37% 22.89MB 4.36%
github.com/syncthing/syncthing/lib/fs.listXattr
22.89MB 4.36% 81.73% 22.89MB 4.36%
github.com/syncthing/syncthing/lib/protocol.(*PlatformData).SetXattrs
15.35MB 2.92% 84.66% 15.36MB 2.93%
github.com/syndtr/goleveldb/leveldb/util.(*BufferPool).Get
	   15.28MB  2.91% 87.57%    15.28MB  2.91%  strings.(*Builder).grow

(The usage for xattrs is reduced from 976 MB to 68 MB)
2023-12-04 12:48:17 +01:00
.github build: Ursrv image for infrastructure 2023-11-15 08:48:00 +01:00
assets assets, gui: Losslessly compress all JPG, PNG, and PDF images (#6265) 2020-01-16 13:52:43 +01:00
cmd cmd/stcrashreceiver: Add metrics for diskstore inventory 2023-11-27 08:24:59 +01:00
etc etc: Update sysctl configs (#8955) 2023-06-28 07:05:12 +02:00
gui gui, man, authors: Update docs, translations, and contributors 2023-12-04 03:45:22 +00:00
lib lib/fs: Reduce memory usage in xattrs handling (#9251) 2023-12-04 12:48:17 +01:00
man gui, man, authors: Update docs, translations, and contributors 2023-12-04 03:45:22 +00:00
meta build: Add more GitHub Actions 2023-02-22 10:56:55 +01:00
next-gen-gui lib/api: Add cache busting for basic auth (ref #9208) (#9215) 2023-11-14 11:57:39 +01:00
proto Add HTML login form (fixes #4137) (#8757) 2023-10-06 13:00:58 +02:00
script build: Support new nested namespaces in Weblate downloads 2023-11-20 09:47:08 +01:00
test lib/api: Add cache busting for basic auth (ref #9208) (#9215) 2023-11-14 11:57:39 +01:00
.codecov.yml build: Add test coverage info (#7502) 2021-04-05 10:25:39 +02:00
.deepsource.toml build: Fix deepsource test & exclude patterns (#7969) 2021-09-26 12:08:59 +02:00
.gitattributes lib/protocol: Revert protobuf encoder changes in v0.14.17 (fixes #3855) 2017-01-01 17:19:00 +00:00
.gitignore gitignore: All exe files, no editor configs (#9126) 2023-09-25 14:17:01 +00:00
.golangci.yml golangci: Skip godox 2019-12-18 11:33:36 +01:00
.yamlfmt build: Add more GitHub Actions 2023-02-22 10:56:55 +01:00
AUTHORS gui, man, authors: Update docs, translations, and contributors 2023-11-20 08:49:38 +00:00
build.go cmd/ursrv: Merge ursrv and uraggregate as subcommands 2023-07-10 09:00:57 +02:00
build.ps1 build: Clean up build.sh, add build.ps1 (#6689) 2020-05-28 12:42:15 +02:00
build.sh gui: Switch to Weblate for translations (#8777) 2023-02-07 12:22:08 +01:00
CONDUCT.md conduct: Upgrade to Contributor Covenant 2018-06-20 23:53:06 +02:00
CONTRIBUTING.md gui: Switch to Weblate for translations (#8777) 2023-02-07 12:22:08 +01:00
Dockerfile build: Fixup Docker changes from previous (#9223) 2023-11-14 08:17:34 +01:00
Dockerfile.builder build: Fixup Docker changes from previous (#9223) 2023-11-14 08:17:34 +01:00
Dockerfile.stcrashreceiver build: Fixup Docker changes from previous (#9223) 2023-11-14 08:17:34 +01:00
Dockerfile.stdiscosrv build: Fixup Docker changes from previous (#9223) 2023-11-14 08:17:34 +01:00
Dockerfile.strelaypoolsrv build: Fixup Docker changes from previous (#9223) 2023-11-14 08:17:34 +01:00
Dockerfile.strelaysrv build: Fixup Docker changes from previous (#9223) 2023-11-14 08:17:34 +01:00
Dockerfile.stupgrades build: Fixup Docker changes from previous (#9223) 2023-11-14 08:17:34 +01:00
Dockerfile.ursrv build: Ursrv image for infrastructure 2023-11-15 08:48:00 +01:00
go.mod build: Update dependencies 2023-11-14 07:41:53 +01:00
go.sum build: Update dependencies 2023-11-14 07:41:53 +01:00
GOALS.md readme: Style fixes, add security note (#9136) 2023-09-28 11:55:48 +02:00
LICENSE all: Update license url to https (ref #3976) 2017-02-09 08:04:16 +01:00
README-Docker.md docker: Add env var to control capabilities (#8552) 2022-09-26 13:39:41 +02:00
README.md readme: Style fixes, add security note (#9136) 2023-09-28 11:55:48 +02:00
tools.go build: Update quic-go for Go 1.19 (#8483) 2022-08-03 15:43:26 +02:00

Syncthing


MPLv2 License CII Best Practices Go Report Card

Goals

Syncthing is a continuous file synchronization program. It synchronizes files between two or more computers. We strive to fulfill the goals below. The goals are listed in order of importance, the most important ones first. This is the summary version of the goal list - for more commentary, see the full Goals document.

Syncthing should be:

  1. Safe From Data Loss

    Protecting the user's data is paramount. We take every reasonable precaution to avoid corrupting the user's files.

  2. Secure Against Attackers

    Again, protecting the user's data is paramount. Regardless of our other goals, we must never allow the user's data to be susceptible to eavesdropping or modification by unauthorized parties.

  3. Easy to Use

    Syncthing should be approachable, understandable, and inclusive.

  4. Automatic

    User interaction should be required only when absolutely necessary.

  5. Universally Available

    Syncthing should run on every common computer. We are mindful that the latest technology is not always available to every individual.

  6. For Individuals

    Syncthing is primarily about empowering the individual user with safe, secure, and easy to use file synchronization.

  7. Everything Else

    There are many things we care about that don't make it on to the list. It is fine to optimize for these values, as long as they are not in conflict with the stated goals above.

Getting Started

Take a look at the getting started guide.

There are a few examples for keeping Syncthing running in the background on your system in the etc directory. There are also several GUI implementations for Windows, Mac, and Linux.

Docker

To run Syncthing in Docker, see the Docker README.

Vote on features/bugs

We'd like to encourage you to vote on issues that matter to you. This helps the team understand what are the biggest pain points for our users, and could potentially influence what is being worked on next.

Getting in Touch

The first and best point of contact is the Forum. If you've found something that is clearly a bug, feel free to report it in the GitHub issue tracker.

If you believe that youve found a Syncthing-related security vulnerability, please report it by emailing security@syncthing.net. Do not report it in the Forum or issue tracker.

Building

Building Syncthing from source is easy. After extracting the source bundle from a release or checking out git, you just need to run go run build.go and the binaries are created in ./bin. There's a guide with more details on the build process.

Signed Releases

As of v0.10.15 and onwards, release binaries are GPG signed with the key D26E6ED000654A3E, available from https://syncthing.net/security.html and most key servers.

There is also a built-in automatic upgrade mechanism (disabled in some distribution channels) which uses a compiled in ECDSA signature. macOS binaries are also properly code signed.

Documentation

Please see the Syncthing documentation site [source].

All code is licensed under the MPLv2 License.