octoleo/restic
2
2
Fork 0
mirror of https://github.com/octoleo/restic.git synced 2024-11-27 07:16:40 +00:00
Code Issues Releases Wiki Activity
1,743 Commits 7 Branches 33 Tags 66 MiB
223dc78acb
Commit Graph

308 Commits

Author SHA1 Message Date
Alexander Neumann
e9cddc0be5 Fix TestFindUsedBlobs 2016-08-16 21:28:55 +02:00
Alexander Neumann
d7e5f11b78 Export FindUsedBlobs 2016-08-16 21:28:55 +02:00
Alexander Neumann
2b1b6d8c2a Export ListAllPacks 2016-08-16 21:28:55 +02:00
Alexander Neumann
acc2fa5816 Fix TestRepack 
* Decrease number of blobs for use in test
 * Fail the test when there's a duplicate blob
 2016-08-16 21:28:54 +02:00
Alexander Neumann
6285f31604 Use pack.BlobSet instead of backend.IDSet 2016-08-16 21:28:54 +02:00
Alexander Neumann
3cca831b2e Fix invalid type in newly created packs 2016-08-16 21:28:54 +02:00
Alexander Neumann
cff6fea32a Fix 'cat' command 2016-08-16 21:28:54 +02:00
Alexander Neumann
17e1872544 Switch order of parameters to repo.LoadBlob() 2016-08-16 21:28:54 +02:00
Alexander Neumann
246302375d Index: Add multiple packs per blob, pack.Type 
Change the index so that a blob can be contained in multiple packs.

Require passing the blob type to all lookup functions.
 2016-08-16 21:28:54 +02:00
Alexander Neumann
231da4ff80 Remove old repacking code 2016-08-16 21:28:54 +02:00
Alexander Neumann
1b4b469440 Add pack.Handle and pack.Handles 2016-08-16 21:28:54 +02:00
Alexander Neumann
35e3762e37 Remove dead code 2016-08-16 21:28:54 +02:00
Alexander Neumann
7e732dbd2d Allow multiple entries in the index 2016-08-16 21:28:54 +02:00
Alexander Neumann
8b4d4ec25f Fix TestCreateSnapshot, do not store duplicate data 2016-08-16 21:28:54 +02:00
Alexander Neumann
035d0aeb31 Do not create duplicate content for tests 2016-08-16 21:28:54 +02:00
Alexander Neumann
f1bc181c5b Add more checks for tests 2016-08-16 21:28:54 +02:00
Alexander Neumann
50b724ca23 Fix stylistic issues with FindUsedBlobs 2016-08-16 21:28:54 +02:00
Alexander Neumann
6227821b4e Move functions to correct file 2016-08-16 21:28:54 +02:00
Alexander Neumann
810056c2bc Correct packages for tests 2016-08-16 21:28:54 +02:00
Alexander Neumann
34b3e3a095 Split index/repack functions to different files 2016-08-16 21:28:54 +02:00
Alexander Neumann
bdd085e9f1 Prevent loops when finding used blobs 2016-08-16 21:28:54 +02:00
Alexander Neumann
ffc3503e6f Add first version of FindUsedBlobs 2016-08-16 21:28:54 +02:00
Alexander Neumann
51b16ad57d Add handy functions to backend.IDSet 2016-08-16 21:28:54 +02:00
Alexander Neumann
723592d923 Move FindUsedBlobs to package restic 2016-08-16 21:28:54 +02:00
Alexander Neumann
22aa17091b Add test for FindUsedBlobs 2016-08-16 21:28:54 +02:00
Alexander Neumann
4720a7d807 Allow specifying chunker polynomial for tests 2016-08-16 21:28:54 +02:00
Alexander Neumann
d5323223f4 Change repository Init() function to allow better testing 2016-08-16 21:28:54 +02:00
Alexander Neumann
fe79177b40 Make TestCreateSnapshot return the snapshot itself 2016-08-16 21:28:54 +02:00
Alexander Neumann
5c32ae15c2 Move test checking repo code to checker package 2016-08-16 21:28:54 +02:00
Alexander Neumann
6c2334f505 Make TestCreateSnapshot less verbose 2016-08-16 21:28:54 +02:00
Alexander Neumann
b55ac2afd6 Make test files in test repo less random 2016-08-16 21:28:54 +02:00
Alexander Neumann
d9012b4a64 Add trees recursively to test snapshot 2016-08-16 21:28:54 +02:00
Alexander Neumann
952f124238 Use RandReader instead of rand directly 
This is a fix to be backwards-compatible with Go < 1.6.
 2016-08-16 21:28:54 +02:00
Alexander Neumann
14db71d3fa Move RandReader to repository package 2016-08-16 21:28:54 +02:00
Alexander Neumann
f59ffcaeae Correct comment 2016-08-16 21:28:54 +02:00
Alexander Neumann
d609e4a986 Extended plaintext buffer if necessary 2016-08-16 21:28:54 +02:00
Alexander Neumann
0e6c72ad1d Implement Repack() 2016-08-16 21:28:54 +02:00
Alexander Neumann
d5f42201c5 Fix test for Repack 2016-08-16 21:28:54 +02:00
Alexander Neumann
122a0944a6 Do not repack blobs that shouldn't be kept 2016-08-16 21:28:54 +02:00
Alexander Neumann
fa26ecc8f9 Make rebuild-index use the code in package repository 2016-08-16 21:28:54 +02:00
Alexander Neumann
00139648a0 Implement Repack() 2016-08-16 21:28:54 +02:00
Alexander Neumann
6ba38e9a38 Add tests for Repack() 2016-08-16 21:28:54 +02:00
Jan Stürtz
b108966b12 Fix 567 (#570) 
* Patch for  https://github.com/restic/restic/issues/567
Backup also files on windows with longer pathnames than 255 chars (e.g. from node).

as fd0 says "So, as far as I can see, we need to have custom methods for all functions that accept a path, so that on Windows we can substitute the normal (possibly relative) path used within restic by an (absolute) UNC path, and only then call the underlying functions like os.Stat(), os.Lstat(), os.Open() and so on.

I've already thought about adding a generic abstraction for the file system (so we can mock this easier in tests), and this looks like a good opportunity to build it."

* fixed building tests

* Restructured patches
Add Wrapper for filepath.Walk

* using \\?\ requires absolute pathes to be used.
Now all tests run

* used gofmt on the code

* Restructured Code. No patches dir, integrate the file functions into restic/fs/

There is still an issue, because restic.fs.Open has a different api the os.Open, which returns the result of OpenFile, but takes only a string

* Changed the last os.Open() calls to fs.Open() after extending the File interface

* fixed name-clash of restic.fs and fuse.fs detected by travis

* fixed fmt with gofmt

* c&p failure: removed fixpath() call.

* missing include

* fixed includes in linux variant

* Fix for Linux. Fd() is required on File interface

* done gofmt
 2016-08-15 21:59:13 +02:00
Alexander Neumann
fa4570bde8 Always use forward slashes in file names 2016-08-11 19:41:47 +02:00
Alexander Neumann
f6c2787d80 Use the platform-independent function for joining 2016-08-11 19:37:22 +02:00
Alexander Neumann
4b8b625b90 Merge pull request #562 from damekr/implement-gomaxprocs-env 
Issue-535: restic respect GOMAXPROCS env variable depending on go version
 2016-08-11 19:09:54 +02:00
damekr
be00d91967 Respect GOMAXPROCS variable 
Closes #535
 2016-08-08 21:37:20 +02:00
Alexander Neumann
fc9b27c533 Revert "Fix TestCreateSnapshot, do not generate duplicate data" 
This reverts commit 628fb0fb72.
 2016-08-02 22:11:55 +02:00
Alexander Neumann
628fb0fb72 Fix TestCreateSnapshot, do not generate duplicate data 2016-08-01 22:01:34 +02:00
Alexander Neumann
8418fed18e Handle empty files correctly 2016-07-29 21:18:32 +02:00
Alexander Neumann
3de989b7bb Fix panic with empty files 2016-07-29 21:05:36 +02:00
Alexander Neumann
5afda94a3c Handle reads with large offsets 2016-07-29 20:55:09 +02:00
Ben Agricola
edb1843f24 Explicitly Close() obj after ReadFull() 
Signed-off-by: Ben Agricola <bagricola@squiz.co.uk>
 2016-07-29 14:18:02 +01:00
mappu
32985f7904 idset.go: micro-optimise away redundant scan 2016-07-26 09:36:31 +12:00
Alexander Neumann
6b7ddf1b03 Don't report valid types as invalid 
Closes #528
 2016-07-20 20:46:57 +02:00
Alexander Neumann
902f619a06 Fix call to minio.New() 
The last parameter changed semantics from `insecure` to `secure`.
 2016-06-08 21:33:18 +02:00
Gerdus van Zyl
73e9cac5c4 gofmt + small doc fix 2016-05-10 22:20:03 +02:00
Gerdus van Zyl
8010a0d90c fix 2016-05-10 21:57:30 +02:00
Gerdus van Zyl
3cb68ddb0d Add option to restore latest snapshot with optional path and source filters 
eg restic -r r1 restore latest --target restore2 --path "D:\dev\restic\bin\s1"
path and source filters also added to snapshot cmd
eg restic -r r1 snapshots --source nucore --path="D:\dev\restic\bin\s1"

Add option to restore latest snapshot with optional path and source filters

eg restic -r r1 restore latest --target restore2 --path "D:\dev\restic\bin\s1"
path and source filters also added to snapshot cmd
eg restic -r r1 snapshots --source nucore --path="D:\dev\restic\bin\s1"
 2016-05-10 21:41:26 +02:00
Gerdus van Zyl
49f82f54b0 rebase, change source to host and add description to manual 2016-05-10 21:40:32 +02:00
Alexander Neumann
6bc7a71e55 Merge pull request #516 from restic/fix-flaky-test 
Fix flaky worker cancel test
 2016-05-09 22:12:39 +02:00
Alexander Neumann
2c1e590e47 Merge pull request #509 from restic/read-from-stdin 
Allow reading data from stdin
 2016-05-09 22:12:22 +02:00
Alexander Neumann
84f7d28abf Merge pull request #515 from viric/fix_traverse_order 
Traverse paths in the same order as parent snapshot
 2016-05-09 22:10:36 +02:00
Alexander Neumann
cb75737770 Merge pull request #514 from viric/fix_parent_search 
Better backup parent snapshot search. Part of #513
 2016-05-09 22:10:32 +02:00
Alexander Neumann
fb45ea139d Add barrier 2016-05-09 21:29:13 +02:00
Alexander Neumann
bce0bbeda2 Add Benchmark for ArchiveReader 2016-05-09 21:16:59 +02:00
Alexander Neumann
c6d934a685 Fix flaky worker cancel test 2016-05-09 20:41:55 +02:00
Alexander Neumann
4146c09a04 Add test for ArchiveReader() 2016-05-09 20:11:32 +02:00
Alexander Neumann
43f9c2d36e backup: Save file size when reading from stdin 2016-05-09 20:11:32 +02:00
Alexander Neumann
5e0813ca04 fuse: Use correct file size in case it's zero 2016-05-09 20:11:32 +02:00
Alexander Neumann
6ee9baa9c5 fuse: Add debug logs 2016-05-09 20:11:32 +02:00
Alexander Neumann
7c76ff3aaf Allow reading backups from stdin 2016-05-09 20:11:32 +02:00
Lluís Batlle i Rossell
4818a8e356 Fix gofmt 2016-05-09 16:31:59 +02:00
Lluís Batlle i Rossell
83aa63365a Not exporting baseNameSlice. Noone else wants it. 2016-05-09 14:46:14 +02:00
Lluís Batlle i Rossell
aed73be93d Improve comment according to hound guidelines 2016-05-09 14:44:03 +02:00
Lluís Batlle i Rossell
4ea62ecbcc Traverse paths in the same order as parent snapshot 
This is the 2nd partial fix to #513.

The archivepipe requires the snapshot paths and the backup paths to be
traversed in the same order, and they were sorted differently: the backup paths
by full path, and the snapshot by basename path.
 2016-05-09 14:32:17 +02:00
Lluís Batlle i Rossell
60c8c90d35 Better backup parent snapshot search. Part of #513 
I look for the newest snapshot that contains all supplied paths to backup.
 2016-05-09 12:42:12 +02:00
Alexander Neumann
20afed4058 Checker: handle symlinks 2016-05-08 23:16:17 +02:00
Alexander Neumann
a2224e380b Address style issues identified by Hound 2016-05-08 22:38:38 +02:00
Alexander Neumann
31030baca3 Add comment 2016-05-08 13:51:33 +02:00
Alexander Neumann
173940cbdf Add repository.ListPack 2016-05-08 13:51:21 +02:00
Alexander Neumann
6fc3590838 Remove repository.SaveFrom() 2016-05-08 13:13:29 +02:00
Alexander Neumann
43f7a1fcd9 Correct log statement 2016-05-08 13:09:36 +02:00
Alexander Neumann
7faf272996 Progress: Use reference to sync.Once 2016-05-08 13:04:58 +02:00
Alexander Neumann
514a43f74b Add more tests 2016-05-08 12:25:01 +02:00
Alexander Neumann
6655511ab8 checker: test file mode 2016-05-08 12:25:01 +02:00
Alexander Neumann
168cfc2f6d Add testing helper functions 2016-05-08 12:25:01 +02:00
Alexander Neumann
6cfa0d502d Add LoadAllSnapshots() 2016-05-08 12:25:01 +02:00
Alexander Neumann
a996dbb9d6 check: Add more checks for nodes 2016-05-08 12:25:01 +02:00
Alexander Neumann
3f8da47a0c Fix restic s3 backend for new minio-go version 2016-05-07 23:38:41 +02:00
Alexander Neumann
4531456be5 Merge pull request #497 from Thor77/excludeFileExpandEnv 
Expand environment-variables in exclude-files
 2016-04-18 21:40:30 +02:00
Alexander Neumann
039019689a Merge pull request #500 from restic/fix-499 
Fix exclude filters with trailing slash
 2016-04-18 21:02:27 +02:00
Alexander Neumann
6fe25548bd Add another filter test 2016-04-17 22:04:42 +02:00
Alexander Neumann
9002eaa259 Fix exclude filters with trailing slash 2016-04-17 21:54:12 +02:00
Alexander Neumann
ead6d11ecf Backend tests: remove debug 2016-04-17 17:39:14 +02:00
Thor77
9644399074 add environment-var expanding for exclude-files 2016-04-16 22:04:29 +02:00
Alexander Neumann
23d7464306 Umount fuse in tests 
This corrects the order when the fuse mount is terminated by closing the
done channel: Before, restic would close the fuse connection and only
afterwards try to remove the mount, that does not work.

Closes #494
 2016-04-13 20:18:54 +02:00
Alexander Neumann
32a5778602 Merge pull request #490 from Thor77/backupExcludeFile 
add backup --exclude-file
 2016-04-06 00:09:46 +02:00
Thor77
1c1eacfc94 add backup --exclude-file 2016-04-01 13:53:22 +02:00
Alexander Neumann
ddfadae6f6 Fix compilation for Go 1.3 2016-03-28 16:09:28 +02:00
Alexander Neumann
c8f46ce81d fs: Require Go1.4 for Linux 2016-03-28 15:48:18 +02:00
Alexander Neumann
c30f4a9134 fs: remove unneeded code 2016-03-28 15:33:10 +02:00
Alexander Neumann
b7713d2d34 local backend: Drop file content from cache after write 2016-03-28 15:31:25 +02:00
Alexander Neumann
5b5bb070b9 fs: Split out ClearCache from File 2016-03-28 15:31:25 +02:00
Alexander Neumann
feb664620a Use fadvise() to not cache the content of files read 2016-03-28 15:26:46 +02:00
Matt Holt
e6ba9e5849 Change ErrNoKeyFound message 
For #438. I was just going to change it to "wrong password" but then I saw that it might actually be the case that no key could be found, so I changed it to what I did. Let me know if you'd like something different!
 2016-03-11 08:21:01 -07:00
Alexander Neumann
e4a6dd8c8c Use newRandReader instead of rand.New() 
This needs to be done since for Go < 1.6 rand.Rand does not implement
io.Reader.
 2016-03-06 14:21:02 +01:00
Alexander Neumann
18c3024171 Unexport NewPackerManager 2016-03-06 14:20:48 +01:00
Alexander Neumann
1e1368eea3 Add randReader for tests 
This can be removed once we require Go 1.6.
 2016-03-06 13:59:06 +01:00
Alexander Neumann
cda7616c82 Remove tempdir for packerManager 2016-03-06 13:14:06 +01:00
Alexander Neumann
015cea0c50 PackerManager: Remove debug comment 2016-03-06 12:35:21 +01:00
Alexander Neumann
c0b5f5a8af Fix all code which uses repository.New() 2016-03-06 12:34:23 +01:00
Alexander Neumann
f956f60f9f PackerManager: use tempfiles instead of memory buffers 2016-03-06 12:26:25 +01:00
Alexander Neumann
f893ec57cb Add test and benchmark for PackerManager 2016-03-05 15:58:39 +01:00
Alexander Neumann
8d1a5731f3 Remove integration test for 'optimize' 2016-02-27 13:38:05 +01:00
Alexander Neumann
04b3ce00e2 Remove command 'optimize' 
It was discovered that the current code may delete still-referenced
blobs, so we'll remove the command for now.

This closes #434
 2016-02-27 13:12:22 +01:00
Alexander Neumann
9386bfbafa checker: Do not use reference in checker errors 2016-02-27 13:10:35 +01:00
Alexander Neumann
a613e23e34 checker: Use backend.IDSet instead of custom struct 2016-02-27 13:10:35 +01:00
Alexander Neumann
5ce1375ddd Rename non-exported function 2016-02-27 13:10:35 +01:00
Alexander Neumann
4cefd456bb Refactor rebuild-index code 
This code reads all pack headers from all packs and rebuilds the index
from scratch. Afterwards, all indexes are removed. This is needed
because in #434 the command `optimize` produced a broken index that
did not contain a blob any more. Running `rebuild-index` should fix
this.
 2016-02-27 13:10:35 +01:00
Alexander Neumann
bc911f4609 cmd_dump: Only load pack header 2016-02-27 13:06:21 +01:00
Alexander Neumann
090920039f pack: Add test with backend.NewReadSeeker 
This uses the new backend ReadSeeker with the unpacker.
 2016-02-27 13:06:21 +01:00
Alexander Neumann
21a99397ff worker: fix tests 
The test failed on Windows, probably because the machine used for CI was
too slow. The new test doesn't depend on timing any more.
 2016-02-27 13:06:21 +01:00
Alexander Neumann
482fc9f51d Add backend.readSeeker 
This struct implements an io.ReadSeeker on top of a backend. This is the
easiest way to make the packer compatible to the new backend without
loading a complete pack into a bytes.Buffer.
 2016-02-27 13:06:21 +01:00
Alexander Neumann
b114ab7108 cmd_dump: Allow dumping all blobs in all packs 
I had the suspicion that one of my repositories contained redundant
data blobs that is not recorder in the index. In order to check this I
needed to dump information about the pack files without consulting the
index. The code here iterates over all packs and dumps the headers, and
dumps information about them in JSON (so we can use other tools to parse
that information again).
 2016-02-27 13:06:21 +01:00
Alexander Neumann
4cb4a3ac7f Add separate goroutine that closes the output chan 
This allows iterating over the output channel without having to start
another Goroutine outside of the worker pool. This also removes the need
for calling Wait().
 2016-02-27 13:06:21 +01:00
Alexander Neumann
ee422110c8 Make worker pools input/output chans symmetric 
Input and output channel are now both of type `chan Job`, this makes it
possible to chain multiple worker pools together.
 2016-02-27 13:06:21 +01:00
Alexander Neumann
e5ee4eba53 Add worker pool 
A worker pool is needed whenever something should be done concurrently.
This small library makes it easy to create a worker pool by specifying
channels, concurrency and a function that should be executed for each
job and returns a result and an error.
 2016-02-27 13:06:21 +01:00
Alexander Neumann
1e0b7dbdd2 Merge pull request #477 from restic/rest-backend 
rest backend: Remove indirection on http.Client
 2016-02-25 22:24:00 +01:00
Alexander Neumann
17f5b524a6 local: Replace matching code with proper Readdir() 2016-02-24 22:43:04 +01:00
Alexander Neumann
4ae16d7661 repository: Use backend.ID to load index 
This commit uses ParallelWorkFuncParseID() to load all indexes and
ignores file names with invalid format.

This fixes #475.
 2016-02-24 22:41:32 +01:00
Alexander Neumann
77d85cee52 Merge pull request #472 from restic/update-chunker 
Update chunker
 2016-02-24 21:25:15 +01:00
Alexander Neumann
2ce49ea0ee Update code to use the new Chunker interface 2016-02-23 23:14:35 +01:00
Fabian Wickborn
6005bd9833 Fix relative paths in integrations tests 2016-02-23 09:21:50 +01:00
Fabian Wickborn
442780f214 Move commands to src/cmds 2016-02-23 07:21:28 +01:00
Alexander Neumann
9c47a8abfc Merge pull request #467 from fawick/master 
Merging restic-server
 2016-02-22 20:48:18 +01:00
Fabian Wickborn
dd5680dab6 restic-server: Create tmp folder 2016-02-22 20:14:11 +01:00
Fabian Wickborn
1cdbc8e1aa restic-server: Fix folder permissions 2016-02-22 20:14:11 +01:00
Fabian Wickborn
e4168fdde5 restic-server: Reduce memory footprint for saving blobs 
Before, the restic-server read the whole blob (up to 8MB) into memory
prior to writing it to disk. Concurrent writes consumed a lot
of memory. This change writes the blob to a tmp file directly and
renames it afterwards in case there where no errors.
 2016-02-22 20:14:11 +01:00
Fabian Wickborn
4749e610af restic-server: Fix content length for HEAD requests 2016-02-22 20:14:11 +01:00
Fabian Wickborn
51d86370a5 Fixes for the PR 
- Removed external dependencies for test
- Prevent building restic-server w/ Go 1.3

Go versions 1.0, 1.1., and 1.2 are going to fail as well, but they
are "excluded" by README.md already.
 2016-02-22 20:13:55 +01:00
Fabian Wickborn
d86c093480 Merged the restic-server by @bchapuis 
Commit ID in fd0/restic-server at time of merge is
07fae00e7ddd8751b150e2ebf0bff8b2871c77ce
 2016-02-22 20:12:50 +01:00
Alexander Neumann
bb7b9ef3fc Merge pull request #466 from ckemper67/sftp-path-clean 
Cleaned up the sftp parsing logic.
 2016-02-22 19:04:56 +01:00
Alexander Neumann
5dd65a5c19 Merge pull request #464 from restic/rest-backend 
Add REST backend
 2016-02-22 18:52:37 +01:00
Christian Kemper
6eb97ca6cc Cleaned up the sftp parsing logic. 
Simplified and cleaned up the sftp parsing logic. Added support to
path.Clean the directory. Added additional tests.
 2016-02-21 10:50:36 -08:00
Christian Kemper
c2716755f1 Added missing handle to the s3.Stat log message output 2016-02-21 09:59:27 -08:00
Alexander Neumann
7087efaa79 rest backend: Remove indirection on http.Client 2016-02-21 17:06:35 +01:00
Alexander Neumann
8ad98e8040 rest backend: Fixes 2016-02-21 16:35:25 +01:00
Alexander Neumann
f7a10a9b9c backend tests: Test accessing config 
This commit adds real testing for accessing the config file with
different names.
 2016-02-21 16:02:13 +01:00
Alexander Neumann
bd621197f8 Add rest backend to ui parser 2016-02-21 15:33:13 +01:00
Alexander Neumann
ec34da2d66 Add rest backend to location 2016-02-21 15:33:13 +01:00
Alexander Neumann
c2348ba768 Add REST backend 
This is a port of the original work by @bchapuis in
https://github.com/restic/restic/pull/253
 2016-02-21 15:33:13 +01:00
Alexander Neumann
625c987d23 Move sftp test 2016-02-20 20:53:40 +01:00
Alexander Neumann
841326d713 Move build.go and run_integration_tests.go to root 2016-02-20 17:31:21 +01:00
Alexander Neumann
7b6629802b Move "doc" to root dir 2016-02-20 17:31:21 +01:00
Alexander Neumann
c0bd660a9e Rename package 
* github.com/restic/restic -> restic
 2016-02-20 17:31:21 +01:00
Alexander Neumann
0a8ef79dad Move top-level files 2016-02-20 17:31:21 +01:00
Alexander Neumann
b63399d606 Move things around for gb 
This moves all restic source files to src/, and all vendored
dependencies to vendor/src.
 2016-02-20 17:31:20 +01:00