This is a new revision of the discovery server. Relevant changes and
non-changes:
- Protocol towards clients is unchanged.
- Recommended large scale design is still to be deployed nehind nginx (I
tested, and it's still a lot faster at terminating TLS).
- Database backend is leveldb again, only. It scales enough, is easy to
setup, and we don't need any backend to take care of.
- Server supports replication. This is a simple TCP channel - protect it
with a firewall when deploying over the internet. (We deploy this within
the same datacenter, and with firewall.) Any incoming client announces
are sent over the replication channel(s) to other peer discosrvs.
Incoming replication changes are applied to the database as if they came
from clients, but without the TLS/certificate overhead.
- Metrics are exposed using the prometheus library, when enabled.
- The database values and replication protocol is protobuf, because JSON
was quite CPU intensive when I tried that and benchmarked it.
- The "Retry-After" value for failed lookups gets slowly increased from
a default of 120 seconds, by 5 seconds for each failed lookup,
independently by each discosrv. This lowers the query load over time for
clients that are never seen. The Retry-After maxes out at 3600 after a
couple of weeks of this increase. The number of failed lookups is
stored in the database, now and then (avoiding making each lookup a
database put).
All in all this means clients can be pointed towards a cluster using
just multiple A / AAAA records to gain both load sharing and redundancy
(if one is down, clients will talk to the remaining ones).
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4648
As of Go 1.8 it's valid to not have a GOPATH set. We ask the Go tool
what the GOPATH seems to be, and if we find ourselves (or a valid copy
of ourselves...) in that location we do not fiddle with the GOPATH.
This adds support for building with the source placed anywhere and no
GOPATH set. The build script handles this by creating a temporary GOPATH
in the system temp dir (or another specified location) and mirroring the
source there before building. The resulting binaries etc still end up in
the same place as usual, meaning at least the "build", "install", "tar",
"zip", "deb", "snap", "test", "vet", "lint", "metalint" and "clean"
commands work without a GOPATH. To this end these commands internally
use fully qualified package paths like
"github.com/syncthing/syncthing/cmd/..." instead of "./cmd/..." like
before.
There is a new command "gopath" that prepares and echoes the directory
of the temporary GOPATH. This can be used to run other non-build go
commands:
export GOPATH=$(go run build.go gopath) // GOPATH is now set
go test -v -race github.com/syncthing/syncthing/cmd/...
There is a new option "-no-build-gopath" that prevents the
check-and-copy step, instead assuming the temporary GOPATH is already
created and up to date. This is a performance optimization for build
servers running multiple builds commands in sequence:
go run build.go gopath // creates a temporary GOPATH
go run build.go -no-build-gopath -goos=... tar // reuses GOPATH
go run build.go -no-build-gopath -goos=... tar // reuses GOPATH
The temporary GOPATH is placed in the system temporary directory
(os.TempDir()) unless overridden by the STTMPDIR variable. It is named
after the hash of the current directory where build.go is run. The
reason for this is that the name should be unique to a source checkout
without risk for conflict, but still persistent between runs of
build.go.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4253
LGTM: AudriusButkevicius, imsodin
This moves a few things from script/ to a new directory meta/, and makes
them real Go tests. These are the authors, copyright, metalint and gofmt
checks. That means that they can now be run by
go test -v ./meta
and optionally filtered by the usual -run thing to go test. Also -short
will cut down on the metalint stuff and exclude the authors check (which
is slow because it runs git lots of times).
Mainly this makes everything easier on things like build servers where
we can now just run tests instead of do a bunch of scripting.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4252
Things like the package name "syncthing" was hardcoded, which is not
awesome. With this in place we can build debs called syncthing-discosrv,
syncthing-relaysrv and syncthing-relaypoolsrv. I don't actually intend
to build and publish the relaypoolsrv, but the others can be good. Using
the "syncthing-" prefix to make the obvious "apt-cache search syncthing"
actually show them etc.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4206
LGTM: AudriusButkevicius, calmh
Also tweaks the proto definitions:
- [packed=false] on the block_indexes field to retain compat with
v0.14.16 and earlier.
- Uses the vendored protobuf package in include paths.
And, "build.go setup" will install the vendored protoc-gen-gogofast.
This should ensure that a proto rebuild isn't so dependent on whatever
version of the compiler and package the developer has installed...
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3864
