mirror of
https://github.com/octoleo/restic.git
synced 2024-10-31 19:02:32 +00:00
477 lines
23 KiB
ReStructuredText
477 lines
23 KiB
ReStructuredText
..
|
||
Normally, there are no heading levels assigned to certain characters as the structure is
|
||
determined from the succession of headings. However, this convention is used in Python’s
|
||
Style Guide for documenting which you may follow:
|
||
|
||
# with overline, for parts
|
||
* for chapters
|
||
= for sections
|
||
- for subsections
|
||
^ for subsubsections
|
||
" for paragraphs
|
||
|
||
#########################
|
||
Removing backup snapshots
|
||
#########################
|
||
|
||
All backup space is finite, so restic allows removing old snapshots. This can
|
||
be done either manually (by specifying a snapshot ID to remove) or by using a
|
||
policy that describes which snapshots to forget. For all remove operations, two
|
||
commands need to be called in sequence: ``forget`` to remove snapshots, and
|
||
``prune`` to remove the remaining data that was referenced only by the removed
|
||
snapshots. The latter can be automated with the ``--prune`` option of ``forget``,
|
||
which runs ``prune`` automatically if any snapshots were actually removed.
|
||
|
||
Pruning snapshots can be a time-consuming process, depending on the
|
||
number of snapshots and data to process. During a prune operation, the
|
||
repository is locked and backups cannot be completed. Please plan your
|
||
pruning so that there's time to complete it and it doesn't interfere with
|
||
regular backup runs.
|
||
|
||
It is advisable to run ``restic check`` after pruning, to make sure
|
||
you are alerted, should the internal data structures of the repository
|
||
be damaged.
|
||
|
||
Remove a single snapshot
|
||
************************
|
||
|
||
The command ``snapshots`` can be used to list all snapshots in a
|
||
repository like this:
|
||
|
||
.. code-block:: console
|
||
|
||
$ restic -r /srv/restic-repo snapshots
|
||
enter password for repository:
|
||
ID Date Host Tags Directory
|
||
----------------------------------------------------------------------
|
||
40dc1520 2015-05-08 21:38:30 kasimir /home/user/work
|
||
79766175 2015-05-08 21:40:19 kasimir /home/user/work
|
||
bdbd3439 2015-05-08 21:45:17 luigi /home/art
|
||
590c8fc8 2015-05-08 21:47:38 kazik /srv
|
||
9f0bc19e 2015-05-08 21:46:11 luigi /srv
|
||
|
||
In order to remove the snapshot of ``/home/art``, use the ``forget``
|
||
command and specify the snapshot ID on the command line:
|
||
|
||
.. code-block:: console
|
||
|
||
$ restic -r /srv/restic-repo forget bdbd3439
|
||
enter password for repository:
|
||
removed snapshot bdbd3439
|
||
|
||
Afterwards this snapshot is removed:
|
||
|
||
.. code-block:: console
|
||
|
||
$ restic -r /srv/restic-repo snapshots
|
||
enter password for repository:
|
||
ID Date Host Tags Directory
|
||
----------------------------------------------------------------------
|
||
40dc1520 2015-05-08 21:38:30 kasimir /home/user/work
|
||
79766175 2015-05-08 21:40:19 kasimir /home/user/work
|
||
590c8fc8 2015-05-08 21:47:38 kazik /srv
|
||
9f0bc19e 2015-05-08 21:46:11 luigi /srv
|
||
|
||
But the data that was referenced by files in this snapshot is still
|
||
stored in the repository. To cleanup unreferenced data, the ``prune``
|
||
command must be run:
|
||
|
||
.. code-block:: console
|
||
|
||
$ restic -r /srv/restic-repo prune
|
||
enter password for repository:
|
||
repository 33002c5e opened successfully, password is correct
|
||
loading all snapshots...
|
||
loading indexes...
|
||
finding data that is still in use for 4 snapshots
|
||
[0:00] 100.00% 4 / 4 snapshots
|
||
searching used packs...
|
||
collecting packs for deletion and repacking
|
||
[0:00] 100.00% 5 / 5 packs processed
|
||
|
||
to repack: 69 blobs / 1.078 MiB
|
||
this removes: 67 blobs / 1.047 MiB
|
||
to delete: 7 blobs / 25.726 KiB
|
||
total prune: 74 blobs / 1.072 MiB
|
||
remaining: 16 blobs / 38.003 KiB
|
||
unused size after prune: 0 B (0.00% of remaining size)
|
||
|
||
repacking packs
|
||
[0:00] 100.00% 2 / 2 packs repacked
|
||
rebuilding index
|
||
[0:00] 100.00% 3 / 3 packs processed
|
||
deleting obsolete index files
|
||
[0:00] 100.00% 3 / 3 files deleted
|
||
removing 3 old packs
|
||
[0:00] 100.00% 3 / 3 files deleted
|
||
done
|
||
|
||
Afterwards the repository is smaller.
|
||
|
||
You can automate this two-step process by using the ``--prune`` switch
|
||
to ``forget``:
|
||
|
||
.. code-block:: console
|
||
|
||
$ restic forget --keep-last 1 --prune
|
||
snapshots for host mopped, directories /home/user/work:
|
||
|
||
keep 1 snapshots:
|
||
ID Date Host Tags Directory
|
||
----------------------------------------------------------------------
|
||
4bba301e 2017-02-21 10:49:18 mopped /home/user/work
|
||
|
||
remove 1 snapshots:
|
||
ID Date Host Tags Directory
|
||
----------------------------------------------------------------------
|
||
8c02b94b 2017-02-21 10:48:33 mopped /home/user/work
|
||
|
||
1 snapshots have been removed, running prune
|
||
loading all snapshots...
|
||
loading indexes...
|
||
finding data that is still in use for 1 snapshots
|
||
[0:00] 100.00% 1 / 1 snapshots
|
||
searching used packs...
|
||
collecting packs for deletion and repacking
|
||
[0:00] 100.00% 5 / 5 packs processed
|
||
|
||
to repack: 69 blobs / 1.078 MiB
|
||
this removes 67 blobs / 1.047 MiB
|
||
to delete: 7 blobs / 25.726 KiB
|
||
total prune: 74 blobs / 1.072 MiB
|
||
remaining: 16 blobs / 38.003 KiB
|
||
unused size after prune: 0 B (0.00% of remaining size)
|
||
|
||
repacking packs
|
||
[0:00] 100.00% 2 / 2 packs repacked
|
||
rebuilding index
|
||
[0:00] 100.00% 3 / 3 packs processed
|
||
deleting obsolete index files
|
||
[0:00] 100.00% 3 / 3 files deleted
|
||
removing 3 old packs
|
||
[0:00] 100.00% 3 / 3 files deleted
|
||
done
|
||
|
||
Removing snapshots according to a policy
|
||
****************************************
|
||
|
||
Removing snapshots manually is tedious and error-prone, therefore restic allows
|
||
specifying a policy (one or more ``--keep-*`` options) for which snapshots to
|
||
keep. You can for example define how many hourly, daily, weekly, monthly and
|
||
yearly snapshots to keep, and any other snapshots will be removed.
|
||
|
||
.. warning:: If you use an append-only repository with policy-based snapshot
|
||
removal, some security considerations are important. Please refer to the
|
||
section below for more information.
|
||
|
||
.. note:: You can always use the ``--dry-run`` option of the ``forget`` command,
|
||
which instructs restic to not remove anything but instead just print what
|
||
actions would be performed.
|
||
|
||
The ``forget`` command accepts the following policy options:
|
||
|
||
- ``--keep-last n`` keep the ``n`` last (most recent) snapshots.
|
||
- ``--keep-hourly n`` for the last ``n`` hours which have one or more
|
||
snapshots, keep only the most recent one for each hour.
|
||
- ``--keep-daily n`` for the last ``n`` days which have one or more
|
||
snapshots, keep only the most recent one for each day.
|
||
- ``--keep-weekly n`` for the last ``n`` weeks which have one or more
|
||
snapshots, keep only the most recent one for each week.
|
||
- ``--keep-monthly n`` for the last ``n`` months which have one or more
|
||
snapshots, keep only the most recent one for each month.
|
||
- ``--keep-yearly n`` for the last ``n`` years which have one or more
|
||
snapshots, keep only the most recent one for each year.
|
||
- ``--keep-tag`` keep all snapshots which have all tags specified by
|
||
this option (can be specified multiple times).
|
||
- ``--keep-within duration`` keep all snapshots having a timestamp within
|
||
the specified duration of the latest snapshot, where ``duration`` is a
|
||
number of years, months, days, and hours. E.g. ``2y5m7d3h`` will keep all
|
||
snapshots made in the two years, five months, seven days and three hours
|
||
before the latest (most recent) snapshot.
|
||
- ``--keep-within-hourly duration`` keep all hourly snapshots made within the
|
||
specified duration of the latest snapshot. The ``duration`` is specified in
|
||
the same way as for ``--keep-within`` and the method for determining hourly
|
||
snapshots is the same as for ``--keep-hourly``.
|
||
- ``--keep-within-daily duration`` keep all daily snapshots made within the
|
||
specified duration of the latest snapshot.
|
||
- ``--keep-within-weekly duration`` keep all weekly snapshots made within the
|
||
specified duration of the latest snapshot.
|
||
- ``--keep-within-monthly duration`` keep all monthly snapshots made within the
|
||
specified duration of the latest snapshot.
|
||
- ``--keep-within-yearly duration`` keep all yearly snapshots made within the
|
||
specified duration of the latest snapshot.
|
||
|
||
.. note:: All calendar related options (``--keep-{hourly,daily,...}``) work on
|
||
natural time boundaries and *not* relative to when you run ``forget``. Weeks
|
||
are Monday 00:00 to Sunday 23:59, days 00:00 to 23:59, hours :00 to :59, etc.
|
||
They also only count hours/days/weeks/etc which have one or more snapshots.
|
||
|
||
.. note:: All duration related options (``--keep-{within,-*}``) ignore snapshots
|
||
with a timestamp in the future (relative to when the ``forget`` command is
|
||
run) and these snapshots will hence not be removed.
|
||
|
||
.. note:: Specifying ``--keep-tag ''`` will match untagged snapshots only.
|
||
|
||
When ``forget`` is run with a policy, restic first loads the list of all snapshots
|
||
and groups them by their host name and paths. The grouping options can be set with
|
||
``--group-by``, e.g. using ``--group-by paths,tags`` to instead group snapshots by
|
||
paths and tags. The policy is then applied to each group of snapshots individually.
|
||
This is a safety feature to prevent accidental removal of unrelated backup sets. To
|
||
disable grouping and apply the policy to all snapshots regardless of their host,
|
||
paths and tags, use ``--group-by ''`` (that is, an empty value to ``--group-by``).
|
||
|
||
Additionally, you can restrict the policy to only process snapshots which have a
|
||
particular hostname with the ``--host`` parameter, or tags with the ``--tag``
|
||
option. When multiple tags are specified, only the snapshots which have all the
|
||
tags are considered. For example, the following command removes all but the
|
||
latest snapshot of all snapshots that have the tag ``foo``:
|
||
|
||
.. code-block:: console
|
||
|
||
$ restic forget --tag foo --keep-last 1
|
||
|
||
This command removes all but the last snapshot of all snapshots that have
|
||
either the ``foo`` or ``bar`` tag set:
|
||
|
||
.. code-block:: console
|
||
|
||
$ restic forget --tag foo --tag bar --keep-last 1
|
||
|
||
To only keep the last snapshot of all snapshots with both the tag ``foo`` and
|
||
``bar`` set use:
|
||
|
||
.. code-block:: console
|
||
|
||
$ restic forget --tag foo,bar --keep-last 1
|
||
|
||
To ensure only untagged snapshots are considered, specify the empty string '' as
|
||
the tag.
|
||
|
||
.. code-block:: console
|
||
|
||
$ restic forget --tag '' --keep-last 1
|
||
|
||
Let's look at a simple example: Suppose you have only made one backup every
|
||
Sunday for 12 weeks:
|
||
|
||
.. code-block:: console
|
||
|
||
$ restic snapshots
|
||
repository f00c6e2a opened successfully, password is correct
|
||
ID Time Host Tags Paths
|
||
---------------------------------------------------------------
|
||
0a1f9759 2019-09-01 11:00:00 mopped /home/user/work
|
||
46cfe4d5 2019-09-08 11:00:00 mopped /home/user/work
|
||
f6b1f037 2019-09-15 11:00:00 mopped /home/user/work
|
||
eb430a5d 2019-09-22 11:00:00 mopped /home/user/work
|
||
8cf1cb9a 2019-09-29 11:00:00 mopped /home/user/work
|
||
5d33b116 2019-10-06 11:00:00 mopped /home/user/work
|
||
b9553125 2019-10-13 11:00:00 mopped /home/user/work
|
||
e1a7b58b 2019-10-20 11:00:00 mopped /home/user/work
|
||
8f8018c0 2019-10-27 11:00:00 mopped /home/user/work
|
||
59403279 2019-11-03 11:00:00 mopped /home/user/work
|
||
dfee9fb4 2019-11-10 11:00:00 mopped /home/user/work
|
||
e1ae2f40 2019-11-17 11:00:00 mopped /home/user/work
|
||
---------------------------------------------------------------
|
||
12 snapshots
|
||
|
||
Then ``forget --keep-daily 4`` will keep the last four snapshots, for the last
|
||
four Sundays, and remove the other snapshots:
|
||
|
||
.. code-block:: console
|
||
|
||
$ restic forget --keep-daily 4 --dry-run
|
||
repository f00c6e2a opened successfully, password is correct
|
||
Applying Policy: keep the last 4 daily snapshots
|
||
keep 4 snapshots:
|
||
ID Time Host Tags Reasons Paths
|
||
-------------------------------------------------------------------------------
|
||
8f8018c0 2019-10-27 11:00:00 mopped daily snapshot /home/user/work
|
||
59403279 2019-11-03 11:00:00 mopped daily snapshot /home/user/work
|
||
dfee9fb4 2019-11-10 11:00:00 mopped daily snapshot /home/user/work
|
||
e1ae2f40 2019-11-17 11:00:00 mopped daily snapshot /home/user/work
|
||
-------------------------------------------------------------------------------
|
||
4 snapshots
|
||
|
||
remove 8 snapshots:
|
||
ID Time Host Tags Paths
|
||
---------------------------------------------------------------
|
||
0a1f9759 2019-09-01 11:00:00 mopped /home/user/work
|
||
46cfe4d5 2019-09-08 11:00:00 mopped /home/user/work
|
||
f6b1f037 2019-09-15 11:00:00 mopped /home/user/work
|
||
eb430a5d 2019-09-22 11:00:00 mopped /home/user/work
|
||
8cf1cb9a 2019-09-29 11:00:00 mopped /home/user/work
|
||
5d33b116 2019-10-06 11:00:00 mopped /home/user/work
|
||
b9553125 2019-10-13 11:00:00 mopped /home/user/work
|
||
e1a7b58b 2019-10-20 11:00:00 mopped /home/user/work
|
||
---------------------------------------------------------------
|
||
8 snapshots
|
||
|
||
The processed snapshots are evaluated against all ``--keep-*`` options but a
|
||
snapshot only need to match a single option to be kept (the results are ORed).
|
||
This means that the most recent snapshot on a Sunday would match both hourly,
|
||
daily and weekly ``--keep-*`` options, and possibly more depending on calendar.
|
||
|
||
For example, suppose you make one backup every day for 100 years. Then ``forget
|
||
--keep-daily 7 --keep-weekly 5 --keep-monthly 12 --keep-yearly 75`` would keep
|
||
the most recent 7 daily snapshots and 4 last-day-of-the-week ones (since the 7
|
||
dailies already include 1 weekly). Additionally, 12 or 11 last-day-of-the-month
|
||
snapshots will be kept (depending on whether one of them ends up being the same
|
||
as a daily or weekly). And finally 75 or 74 last-day-of-the-year snapshots are
|
||
kept, depending on whether one of them ends up being the same as an already kept
|
||
snapshot. All other snapshots are removed.
|
||
|
||
You might want to maintain the same policy as in the example above, but have
|
||
irregular backups. For example, the 7 snapshots specified with ``--keep-daily 7``
|
||
might be spread over a longer period. If what you want is to keep daily
|
||
snapshots for the last week, weekly for the last month, monthly for the last
|
||
year and yearly for the last 75 years, you can instead specify ``forget
|
||
--keep-within-daily 7d --keep-within-weekly 1m --keep-within-monthly 1y
|
||
--keep-within-yearly 75y`` (note that `1w` is not a recognized duration, so
|
||
you will have to specify `7d` instead).
|
||
|
||
For safety reasons, restic refuses to act on an "empty" policy. For example,
|
||
if one were to specify ``--keep-last 0`` to forget *all* snapshots in the
|
||
repository, restic will respond that no snapshots will be removed. To delete
|
||
all snapshots, use ``--keep-last 1`` and then finally remove the last snapshot
|
||
manually (by passing the ID to ``forget``).
|
||
|
||
Security considerations in append-only mode
|
||
===========================================
|
||
|
||
.. note:: TL;DR: With append-only repositories, one should specifically use the
|
||
``--keep-within`` option of the ``forget`` command when removing snapshots.
|
||
|
||
To prevent a compromised backup client from deleting its backups (for example
|
||
due to a ransomware infection), a repository service/backend can serve the
|
||
repository in a so-called append-only mode. This means that the repository is
|
||
served in such a way that it can only be written to and read from, while delete
|
||
and overwrite operations are denied. Restic's `rest-server`_ features an
|
||
append-only mode, but few other standard backends do. To support append-only
|
||
with such backends, one can use `rclone`_ as a complement in between the backup
|
||
client and the backend service.
|
||
|
||
.. _rest-server: https://github.com/restic/rest-server/
|
||
.. _rclone: https://rclone.org/commands/rclone_serve_restic/
|
||
|
||
To remove snapshots and recover the corresponding disk space, the ``forget``
|
||
and ``prune`` commands require full read, write and delete access to the
|
||
repository. If an attacker has this, the protection offered by append-only
|
||
mode is naturally void. The usual and recommended setup with append-only
|
||
repositories is therefore to use a separate and well-secured client whenever
|
||
full access to the repository is needed, e.g. for administrative tasks such
|
||
as running ``forget``, ``prune`` and other maintenance commands.
|
||
|
||
However, even with append-only mode active and a separate, well-secured client
|
||
used for administrative tasks, an attacker who is able to add garbage snapshots
|
||
to the repository could bring the snapshot list into a state where all the
|
||
legitimate snapshots risk being deleted by an unsuspecting administrator that
|
||
runs the ``forget`` command with certain ``--keep-*`` options, leaving only the
|
||
attacker's useless snapshots.
|
||
|
||
For example, if the ``forget`` policy is to keep three weekly snapshots, and
|
||
the attacker adds an empty snapshot for each of the last three weeks, all with
|
||
a timestamp (see the ``backup`` command's ``--time`` option) slightly more
|
||
recent than the existing snapshots (but still within the target week), then the
|
||
next time the repository administrator (or a scheduled job) runs the ``forget``
|
||
command with this policy, the legitimate snapshots will be removed (since the
|
||
policy will keep only the most recent snapshot within each week). Even without
|
||
running ``prune``, recovering data would be messy and some metadata lost.
|
||
|
||
To avoid this, ``forget`` policies applied to append-only repositories should
|
||
use the ``--keep-within`` option, as this will keep not only the attacker's
|
||
snapshots but also the legitimate ones. Assuming the system time is correctly
|
||
set when ``forget`` runs, this will allow the administrator to notice problems
|
||
with the backup or the compromised host (e.g. by seeing more snapshots than
|
||
usual or snapshots with suspicious timestamps). This is, of course, limited to
|
||
the specified duration: if ``forget --keep-within 7d`` is run 8 days after the
|
||
last good snapshot, then the attacker can still use that opportunity to remove
|
||
all legitimate snapshots.
|
||
|
||
.. _customize-pruning:
|
||
|
||
Customize pruning
|
||
*****************
|
||
|
||
To understand the custom options, we first explain how the pruning process works:
|
||
|
||
1. All snapshots and directories within snapshots are scanned to determine
|
||
which data is still in use.
|
||
2. For all files in the repository, restic finds out if the file is fully
|
||
used, partly used or completely unused.
|
||
3. Completely unused files are marked for deletion. Fully used files are kept.
|
||
A partially used file is either kept or marked for repacking depending on user
|
||
options.
|
||
|
||
Note that for repacking, restic must download the file from the repository
|
||
storage and re-upload the needed data in the repository. This can be very
|
||
time-consuming for remote repositories.
|
||
4. After deciding what to do, ``prune`` will actually perform the repack, modify
|
||
the index according to the changes and delete the obsolete files.
|
||
|
||
The ``prune`` command accepts the following options:
|
||
|
||
- ``--max-unused limit`` allow unused data up to the specified limit within the repository.
|
||
This allows restic to keep partly used files instead of repacking them.
|
||
|
||
The limit can be specified in several ways:
|
||
|
||
* As an absolute size (e.g. ``200M``). If you want to minimize the space
|
||
used by your repository, pass ``0`` to this option.
|
||
* As a size relative to the total repo size (e.g. ``10%``). This means that
|
||
after prune, at most ``10%`` of the total data stored in the repo may be
|
||
unused data. If the repo after prune has a size of 500MB, then at most
|
||
50MB may be unused.
|
||
* If the string ``unlimited`` is passed, there is no limit for partly
|
||
unused files. This means that as long as some data is still used within
|
||
a file stored in the repo, restic will just leave it there. Use this if
|
||
you want to minimize the time and bandwidth used by the ``prune``
|
||
operation. Note that metadata will still be repacked.
|
||
|
||
Restic tries to repack as little data as possible while still ensuring this
|
||
limit for unused data. The default value is 5%.
|
||
|
||
- ``--max-repack-size size`` if set limits the total size of files to repack.
|
||
As ``prune`` first stores all repacked files and deletes the obsolete files at the end,
|
||
this option might be handy if you expect many files to be repacked and fear to run low
|
||
on storage.
|
||
|
||
- ``--repack-cacheable-only`` if set to true only files which contain
|
||
metadata and would be stored in the cache are repacked. Other pack files are
|
||
not repacked if this option is set. This allows a very fast repacking
|
||
using only cached data. It can, however, imply that the unused data in
|
||
your repository exceeds the value given by ``--max-unused``.
|
||
The default value is false.
|
||
|
||
- ``--dry-run`` only show what ``prune`` would do.
|
||
|
||
- ``--verbose`` increased verbosity shows additional statistics for ``prune``.
|
||
|
||
|
||
Recovering from "no free space" errors
|
||
**************************************
|
||
|
||
In some cases when a repository has grown large enough to fill up all disk space or the
|
||
allocated quota, then ``prune`` might fail to free space. ``prune`` works in such a way
|
||
that a repository remains usable no matter at which point the command is interrupted.
|
||
However, this also means that ``prune`` requires some scratch space to work.
|
||
|
||
In most cases it is sufficient to instruct ``prune`` to use as little scratch space as
|
||
possible by running it as ``prune --max-repack-size 0``. Note that for restic versions
|
||
before 0.13.0 ``prune --max-repack-size 1`` must be used. Obviously, this can only work
|
||
if several snapshots have been removed using ``forget`` before. This then allows the
|
||
``prune`` command to actually remove data from the repository. If the command succeeds,
|
||
but there is still little free space, then remove a few more snapshots and run ``prune`` again.
|
||
|
||
If ``prune`` fails to complete, then ``prune --unsafe-recover-no-free-space SOME-ID``
|
||
is available as a method of last resort. It allows prune to work with little to no free
|
||
space. However, a **failed** ``prune`` run can cause the repository to become
|
||
**temporarily unusable**. Therefore, make sure that you have a stable connection to the
|
||
repository storage, before running this command. In case the command fails, it may become
|
||
necessary to manually remove all files from the `index/` folder of the repository and
|
||
run `rebuild-index` afterwards.
|
||
|
||
To prevent accidental usages of the ``--unsafe-recover-no-free-space`` option it is
|
||
necessary to first run ``prune --unsafe-recover-no-free-space SOME-ID`` and then replace
|
||
``SOME-ID`` with the requested ID.
|