This improves the ignore handling so that directories can be fully
ignored (skipped in the watcher) in more cases. Specifically, where the
previous rule was that any complex `!`-pattern would disable skipping
directories, the new rule is that only matches on patterns *after* such
a `!`-pattern disable skipping. That is, the following now does the
intuitive thing:
```
/foo
/bar
!whatever
*
```
- `/foo/**` and `/bar/**` are completely skipped, since there is no
chance anything underneath them could ever be not-ignored
- `!whatever` toggles the "can't skip directories any more" flag
- Anything that matches `*` can't skip directories, because it's
possible we can have `whatever` match something deeper.
To enable this, some refactoring was necessary:
- The "can skip dirs" flag is now a property of the match result, not of
the pattern set as a whole.
- That meant returning a boolean is not good enough, we need to actually
return the entire `Result` (or, like, two booleans but that seemed
uglier and more annoying to use)
- `ShouldIgnore(string) boolean` went away with
`Match(string).IsIgnored()` being the obvious replacement (API
simplification!)
- The watcher then needed to import the `ignore` package (for the
`Result` type), but `fs` imports the watcher and `ignore` imports `fs`.
That's a cycle, so I broke out `Result` into a package of its own so
that it can be safely imported everywhere in things like `type Matcher
interface { Match(string) result.Result }`. There's a fair amount of
stuttering in `result.Result` and maybe we should go with something like
`ignoreresult.R` or so, leaving this open for discussion.
Tests refactored to suit, I think this change is in fact quite well
covered by the existing ones...
Also some noise because a few of the changed files were quite old and
got the `gofumpt` treatment by my editor. Sorry not sorry.
---------
Co-authored-by: Simon Frei <freisim93@gmail.com>
### Purpose
Fix#9241 by expanding tildes in version paths.
When creating the versioner file system, first try to expand any leading
tildes to the user's home directory before handling relative paths. This
makes a version path `"~/p"` expand to `"$HOME/p"` instead of
`"/folder/~/p"`.
### Testing
Added a test to lib/versioner that exercises this code path. Also
manually tested with local syncthing instances.
### Purpose
This PR changes behaviour of syncthing related to `receive-only`
folders, which I believe to be a bug since I wouldn't expect the current
behaviour. With the current syncthing codebase, a file of a
`receive-only` folder that is only modified locally can cause the
creation of a `.sync-conflict` file.
### Testing
Consider this szenario: Setup two paired clients that sync a folder with
a given file (e.g. `Test.txt`). One of the clients configures the folder
to be `receive-only`. Now, change the contents of the file for the
receive-only client **_twice_**.
With the current syncthing codebase, this leads to the creation of a
`.sync-conflict` file that contains the modified contents, while the
regular `Test.txt` file is reset to the cluster's provided contents.
This is due to a `protocol.FileInfo#ShouldConflict` check, that is
succeeding on the locally modified file.
This PR changes this behaviour to not reset the file and not cause the
creation of a `.sync-conflict`. Instead, the second content update is
treated the same as the first content update.
This PR also contains a test that fails on the current codebase and
succeeds with the changes introduced in this PR.
### Screenshots
This is not a GUI change
### Documentation
This is not a user visible change.
## Authorship
Your name and email will be added automatically to the AUTHORS file
based on the commit metadata.
#### Thanks to all the syncthing folks for this awesome piece of
software!
### Purpose
This implements CLI completion using the Kongplete module. As a side
effect a CLI structure for syncthing/cli was created for kongplete to be
able to parse and implement CLI completion.
### Testing
I've tested the autocompletion manually, and it had worked, but I hadn't
added any tests so as to test it automatically. Additionally, I ran `go
run build.go test` with all tests passing.
This adds a "token manager" which handles storing and checking expired
tokens, used for both sessions and CSRF tokens. It removes the old,
corresponding functionality for CSRFs which saved things in a file. The
result is less crap in the state directory, and active login sessions
now survive a Syncthing restart (this really annoyed me).
It also adds a boolean on login to create a longer-lived session cookie,
which is now possible and useful. Thus we can remain logged in over
browser restarts, which was also annoying... :)
<img width="1001" alt="Screenshot 2023-12-12 at 09 56 34"
src="https://github.com/syncthing/syncthing/assets/125426/55cb20c8-78fc-453e-825d-655b94c8623b">
Best viewed with whitespace-insensitive diff, as a bunch of the auth
functions became methods instead of closures which changed indentation.
```
% export GOTOOLCHAIN=go1.20.7
% go list -m all | cut -d ' ' -f 1 | xargs go get -u
% go mod tidy
```
Except:
- github.com/jackpal/gateway now requires Go 1.21
- github.com/shirou/gopsutil breaks linux-mips in latest version
Currently, with a large number of versioned files, there is a delay
between the Restore Versions modal showing up on the screen and
initialisation of the actual versions tree. This leads to a situation,
where the modal is initially empty, which confuses the user, making
them think that something is not working correctly.
To avoid the above, always show the loading data information. The string
is displayed using static HTML first, and then replaced with the exact
same content once the tree has been initialised. Both elements use the
same style and position, so there is no visual shift between the two.
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
Bumps
[actions/download-artifact](https://github.com/actions/download-artifact)
from 3 to 4.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/actions/download-artifact/releases">actions/download-artifact's
releases</a>.</em></p>
<blockquote>
<h2>v4.0.0</h2>
<h2>What's Changed</h2>
<p>The release of upload-artifact@v4 and download-artifact@v4 are major
changes to the backend architecture of Artifacts. They have numerous
performance and behavioral improvements.</p>
<p>For more information, see the <a
href="https://github.com/actions/toolkit/tree/main/packages/artifact"><code>@actions/artifact</code></a>
documentation.</p>
<h2>New Contributors</h2>
<ul>
<li><a href="https://github.com/bflad"><code>@bflad</code></a> made
their first contribution in <a
href="https://redirect.github.com/actions/download-artifact/pull/194">actions/download-artifact#194</a></li>
</ul>
<p><strong>Full Changelog</strong>: <a
href="https://github.com/actions/download-artifact/compare/v3...v4.0.0">https://github.com/actions/download-artifact/compare/v3...v4.0.0</a></p>
<h2>v3.0.2</h2>
<ul>
<li>Bump <code>@actions/artifact</code> to v1.1.1 - <a
href="https://redirect.github.com/actions/download-artifact/pull/195">actions/download-artifact#195</a></li>
<li>Fixed a bug in Node16 where if an HTTP download finished too quickly
(<1ms, e.g. when it's mocked) we attempt to delete a temp file that
has not been created yet <a
href="hhttps://redirect.github.com/actions/toolkit/pull/1278">actions/toolkit#1278</a></li>
</ul>
<h2>v3.0.1</h2>
<ul>
<li><a
href="https://redirect.github.com/actions/download-artifact/pull/178">Bump
<code>@actions/core</code> to 1.10.0</a></li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="7a1cd3216c"><code>7a1cd32</code></a>
Merge pull request <a
href="https://redirect.github.com/actions/download-artifact/issues/246">#246</a>
from actions/v4-beta</li>
<li><a
href="8f32874a49"><code>8f32874</code></a>
licensed cache</li>
<li><a
href="b5ff8444b1"><code>b5ff844</code></a>
Merge pull request <a
href="https://redirect.github.com/actions/download-artifact/issues/245">#245</a>
from actions/robherley/v4-documentation</li>
<li><a
href="f07a0f73f5"><code>f07a0f7</code></a>
Update README.md</li>
<li><a
href="7226129829"><code>7226129</code></a>
update test workflow to use different artifact names for matrix</li>
<li><a
href="ada9446619"><code>ada9446</code></a>
update docs and bump <code>@actions/artifact</code></li>
<li><a
href="7eafc8b729"><code>7eafc8b</code></a>
Merge pull request <a
href="https://redirect.github.com/actions/download-artifact/issues/244">#244</a>
from actions/robherley/bump-toolkit</li>
<li><a
href="3132d12662"><code>3132d12</code></a>
consume latest toolkit</li>
<li><a
href="5be1d38671"><code>5be1d38</code></a>
Merge pull request <a
href="https://redirect.github.com/actions/download-artifact/issues/243">#243</a>
from actions/robherley/v4-beta-updates</li>
<li><a
href="465b526e63"><code>465b526</code></a>
consume latest <code>@actions/toolkit</code></li>
<li>Additional commits viewable in <a
href="https://github.com/actions/download-artifact/compare/v3...v4">compare
view</a></li>
</ul>
</details>
<br />
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)
</details>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
If syncOwnership is enabled and the remote uses for example a dockerized
Syncthing it can't fetch the ownername and groupname of the local
instance. Without this patch this led to an endless cycle of detected
changes on the remote and failing re-sync attempts.
This patch skips comparing the ownername and groupname if they zare empty
on one side.
See https://github.com/syncthing/syncthing/issues/9039 for details.
### Testing
Proposed by @calmh in
https://github.com/syncthing/syncthing/issues/9039#issuecomment-1870584783
and tested locally in my setup,
Setup PC 1:
- Syncthing is run in Docker as user `root` and has none of the users
configured that synchronize their files
Setup PC 2:
- this PC has all users locally setup
- Syncthing runs as `systemd` service as user `syncthing` and has
multiple capabilities set to set the correct owner and permissions
Setup PC 3:
- same as PC 2
Handling:
- `PC 1` is send & receive and uses just the `UID` and `GID` identifiers
to store the files
- `PC 2` and `PC 3` synchronize their files over `PC 1` but not directly
to each other
Outcome:
- `PC 2` and `PC 3` should send and receive their files with the correct
ownership and groups from `PC 1`
### Purpose
Instead of hardcoding `SigningKey` as text use `go:embed`. Fixes#9247.
### Testing
* Building syncthing
* Trying to upgrade (signature verification)
On kqueue-systems, folders listen for folder summaries to (be able to)
warn for potential high resource usage. However, it listened for any
folder summary and not for the summary which matches the folder it's
about. This could cause that an unwatched folder causes a folder summary
containing more files than the threshold (10k), and the listening folder
(with the watcher enabled) triggers the warning.
This makes sure that only the folder summaries which are relevant to the
specific folder are being handled.
### Testing
- Fire up some kqueue-system (freebsd, I used).
- add folder A, disable the watcher, add 10001 files
- add folder B with the watcher enabled, no files are needed here
Before the change:
- add an item to folder A, trigger a rescan to speed up the process
- wait some seconds...warning triggered by folder B's
summarySubscription
After the change:
- Only a warning is triggered if the received folder summary matches the
folder which listens for the summaries
Cleanup after #9275.
This renames `fmut` -> `mut`, removes the deadlock detector and
associated plumbing, renames some things from `...PRLocked` to
`...RLocked` and similar, and updates comments.
Apart from the removal of the deadlock detection machinery, no
functional code changes... i.e. almost 100% diff noise, have fun
reviewing.
I'm tired of the fmut/pmut shenanigans. This consolidates both under one
lock; I'm not convinced there are any significant performance
differences with this approach since we're literally just protecting map
juggling...
- The locking goes away when we were already under an appropriate fmut
lock.
- Where we had fmut.RLock()+pmut.Lock() it gets upgraded to an
fmut.Lock().
- Otherwise s/pmut/fmut/.
In order to avoid diff noise for an important change I did not do the
following cleanups, which will be filed in a PR after this one, if
accepted:
- Renaming fmut to just mut
- Renaming methods that refer to being "PRLocked" and stuff like that
- Removing the no longer relevant deadlock detector
- Comments referring to pmut and locking sequences...
`syncthing cli` subcommand was using urfave/cli as the command parser.
This PR replace it with kong, which the main command uses.
Some help texts and error message format are changed. Other than that,
all the command usage and logic remains unchanged.
There's only one place which still uses urfave/cli, which is `syncthing
cli config`, because it uses some magic to dynamically build commands
from struct reflects. I used kong's `passthrough:""` tag to pass any
argument following `syncthing cli config` to urfave/cli parser.
This PR also fixes#9041
---------
Co-authored-by: Jakob Borg <jakob@kastelo.net>
This pull request allows syncthing to request an IPv6
[pinhole](https://en.wikipedia.org/wiki/Firewall_pinhole), addressing
issue #7406. This helps users who prefer to use IPv6 for hosting their
services or are forced to do so because of
[CGNAT](https://en.wikipedia.org/wiki/Carrier-grade_NAT). Otherwise,
such users would have to configure their firewall manually to allow
syncthing traffic to pass through while IPv4 users can use UPnP to take
care of network configuration already.
### Testing
I have tested this in a virtual machine setup with miniupnpd running on
the virtualized router. It successfully added an IPv6 pinhole when used
with IPv6 only, an IPv4 port mapping when used with IPv4 only and both
when dual-stack (IPv4 and IPv6) is used.
Automated tests could be added for SOAP responses from the router but
automatically testing this with a real network is likely infeasible.
### Documentation
https://docs.syncthing.net/users/firewall.html could be updated to
mention the fact that UPnP now works with IPv6, although this change is
more "behind the scenes".
---------
Co-authored-by: Simon Frei <freisim93@gmail.com>
Co-authored-by: bt90 <btom1990@googlemail.com>
Co-authored-by: André Colomb <github.com@andre.colomb.de>
gui: Show folder/device status on small screens
On larger screens, folder and device status is shown in a textual form
directly next to folder and device titles. However, on small screens,
only icons are currently shown, which may be ambiguous to new users, who
cannot possibly know what a specific icon means (see [1]). Thus, on
small screens only, display a new entry in folder/device info that
contains the same textual information that is shown in the title on
larger screens.
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
Co-authored-by: André Colomb <src@andre.colomb.de>
Before introducing the service map and using it for folder runners, the
entries in folderCfgs and folderRunners for the same key/folder were
removed under a single lock. Stopping the folder happens separately
before that with just the read lock. Now with the service map stopping
the folder and removing it from the map is a single operation. And that
still happens with just a read-lock. However even with a full lock it’s
still problematic: After the folder stopped, the runner isn’t present
anymore while the folder-config still is and sais the folder isn't
paused.
The index handler in turn looks at the folder config that is not paused,
thus assumes the runner has to be present -> nil deref on the runner.
A better solution might be to push most of these fmut maps into the
folder - they anyway are needed in there. Then there's just a single
map/source of info that's necessarily consistent. That's quite a bit of
work though, and probably/likely there will be corner cases there too.
This reverts commit e477777f49.
In principle, we could have stayed with `~1.21.1`, but `check-latest:
true` apparently checks some cache/manifest/something that is only
periodically refreshed and isn't aware of 1.21.5 yet. So update the
constraints to force an upgrade.
Also the infrastructure images weren't actually using the constraint
since there was no `setup-go` action...
This reduces allocations, in number and in size, while getting extended
attributes. This is mostly noticable when there is a large number of new
files to scan and we're running with the default scanProgressInterval --
then a queue of files is built in-memory, and this queue includes
extended attributes as part of file metadata. (Arguable it shouldn't,
but that's a more difficult and involved change.)
With 1M files to scan, each with one extended attribute, current peak
memory usage looks like this:
Showing nodes accounting for 1425.30MB, 98.19% of 1451.64MB total
Dropped 1435 nodes (cum <= 7.26MB)
Showing top 10 nodes out of 54
flat flat% sum% cum cum%
976.56MB 67.27% 67.27% 976.56MB 67.27%
github.com/syncthing/syncthing/lib/fs.getXattr
305.44MB 21.04% 88.31% 305.44MB 21.04%
github.com/syncthing/syncthing/lib/scanner.(*walker).walk.func1
45.78MB 3.15% 91.47% 1045.23MB 72.00%
github.com/syncthing/syncthing/lib/fs.(*BasicFilesystem).GetXattr
22.89MB 1.58% 93.04% 22.89MB 1.58%
github.com/syncthing/syncthing/lib/fs.listXattr
22.89MB 1.58% 94.62% 22.89MB 1.58%
github.com/syncthing/syncthing/lib/protocol.(*PlatformData).SetXattrs
16MB 1.10% 95.72% 16.01MB 1.10%
github.com/syndtr/goleveldb/leveldb/memdb.New
After the change, it's this:
Showing nodes accounting for 502.32MB, 95.70% of 524.88MB total
Dropped 1400 nodes (cum <= 2.62MB)
Showing top 10 nodes out of 91
flat flat% sum% cum cum%
305.43MB 58.19% 58.19% 305.43MB 58.19%
github.com/syncthing/syncthing/lib/scanner.(*walker).walk.func1
45.79MB 8.72% 66.91% 68.68MB 13.09%
github.com/syncthing/syncthing/lib/fs.(*BasicFilesystem).GetXattr
32MB 6.10% 73.01% 32.01MB 6.10%
github.com/syndtr/goleveldb/leveldb/memdb.New
22.89MB 4.36% 77.37% 22.89MB 4.36%
github.com/syncthing/syncthing/lib/fs.listXattr
22.89MB 4.36% 81.73% 22.89MB 4.36%
github.com/syncthing/syncthing/lib/protocol.(*PlatformData).SetXattrs
15.35MB 2.92% 84.66% 15.36MB 2.93%
github.com/syndtr/goleveldb/leveldb/util.(*BufferPool).Get
15.28MB 2.91% 87.57% 15.28MB 2.91% strings.(*Builder).grow
(The usage for xattrs is reduced from 976 MB to 68 MB)
LastSeen for a device was only updated when they connected. This now
updates it when they disconnect, so that we remember the last time we
actually saw them. When asking the API for current stats, currently
connected devices get a last seen value of the current time.
