From aafee25d5856105686ff18ccc33d643064a20a12 Mon Sep 17 00:00:00 2001 From: Alexander Neumann Date: Mon, 17 Apr 2017 22:39:00 +0200 Subject: [PATCH] Convert AWS S3 tutorial to rst --- doc/AWS_S3_Tutorial.md | 216 --------------- .../s3 => images/aws_s3}/01_aws_start.png | Bin doc/{img/s3 => images/aws_s3}/02_aws_menu.png | Bin .../aws_s3}/03_buckets_list_before.png | Bin .../aws_s3}/04_bucket_create_start.png | Bin .../aws_s3}/05_bucket_create_review.png | Bin .../aws_s3}/06_buckets_list_after.png | Bin .../s3 => images/aws_s3}/07_iam_start.png | Bin .../s3 => images/aws_s3}/08_user_list.png | Bin .../s3 => images/aws_s3}/09_user_name.png | Bin .../aws_s3}/10_user_pre_policy.png | Bin .../s3 => images/aws_s3}/11_policy_start.png | Bin .../aws_s3}/12_policy_permissions_done.png | Bin .../s3 => images/aws_s3}/13_policy_review.png | Bin .../aws_s3}/14_user_attach_policy.png | Bin .../s3 => images/aws_s3}/15_user_review.png | Bin .../s3 => images/aws_s3}/16_user_created.png | Bin doc/index.rst | 1 + doc/tutorial_aws_s3.rst | 255 ++++++++++++++++++ doc/tutorials.rst | 5 + 20 files changed, 261 insertions(+), 216 deletions(-) delete mode 100644 doc/AWS_S3_Tutorial.md rename doc/{img/s3 => images/aws_s3}/01_aws_start.png (100%) rename doc/{img/s3 => images/aws_s3}/02_aws_menu.png (100%) rename doc/{img/s3 => images/aws_s3}/03_buckets_list_before.png (100%) rename doc/{img/s3 => images/aws_s3}/04_bucket_create_start.png (100%) rename doc/{img/s3 => images/aws_s3}/05_bucket_create_review.png (100%) rename doc/{img/s3 => images/aws_s3}/06_buckets_list_after.png (100%) rename doc/{img/s3 => images/aws_s3}/07_iam_start.png (100%) rename doc/{img/s3 => images/aws_s3}/08_user_list.png (100%) rename doc/{img/s3 => images/aws_s3}/09_user_name.png (100%) rename doc/{img/s3 => images/aws_s3}/10_user_pre_policy.png (100%) rename doc/{img/s3 => images/aws_s3}/11_policy_start.png (100%) rename doc/{img/s3 => images/aws_s3}/12_policy_permissions_done.png (100%) rename doc/{img/s3 => images/aws_s3}/13_policy_review.png (100%) rename doc/{img/s3 => images/aws_s3}/14_user_attach_policy.png (100%) rename doc/{img/s3 => images/aws_s3}/15_user_review.png (100%) rename doc/{img/s3 => images/aws_s3}/16_user_created.png (100%) create mode 100644 doc/tutorial_aws_s3.rst create mode 100644 doc/tutorials.rst diff --git a/doc/AWS_S3_Tutorial.md b/doc/AWS_S3_Tutorial.md deleted file mode 100644 index efc3edf0e..000000000 --- a/doc/AWS_S3_Tutorial.md +++ /dev/null @@ -1,216 +0,0 @@ -# Setting up Restic with Amazon S3 - -## Preface - -This tutorial will show you how to use Restic with AWS S3. It will show you how -to navigate the AWS web interface, create an S3 bucket, create a user with -access to only this bucket, and finally how to connect Restic to this bucket. - -## Prerequisites - -You should already have a `restic` binary available on your system that you can -run. Furthermore, you should also have an account with -[AWS](https://aws.amazon.com/). You will likely need to provide credit card -details for billing purposes, even if you use their -[free-tier](https://aws.amazon.com/free/). - - -## Logging into AWS - -Point your browser to -https://console.aws.amazon.com -and log in using your AWS account. You will be presented with the AWS homepage: -![AWS Homepage](img/s3/01_aws_start.png?raw=true) - -By using the "Services" button in the upper left corder, a menu of all services -provided by AWS can be opened: -![AWS Services Menu](img/s3/02_aws_menu.png?raw=true) - -For this tutorial, the Simple Storage Service (S3), as well as Identity and -Access Management (IAM) are relevant. - - -## Creating the bucket - -First, a bucket to store your backups in must be created. Using the "Services" -menu, navigate to S3. In case you already have some S3 buckets, you will see a -list of them here: -![List of S3 Buckets](img/s3/03_buckets_list_before.png?raw=true) - -Click the "Create bucket" button and choose a name and region for your new -bucket. For the purpose of this tutorial, the bucket will be named -`restic-demo` and reside in Frankfurk. Because the bucket namespace is shared among all AWS users, the -name `restic-demo` may not be available to you. Be creative and choose a unique -bucket name. -![Creating a Bucket](img/s3/04_bucket_create_start.png?raw=true) - -It is not necessary to configure any special properties or permissions of the -bucket just yet. Therefore, just finish the wizard without making any further -changes: -![Reviewing Bucket Creation](img/s3/05_bucket_create_review.png?raw=true) - -The newly created `restic-demo` bucket will no appear on the list of S3 -buckets: -![List With New Bucket](img/s3/06_buckets_list_after.png?raw=true) - - -## Creating a user - -Use the "Services" menu of the AWS web interface to navigate to IAM. This will -bring you to the IAM homepage. To create a new user, click on the "Users" menu -entry on the left: -![IAM Homepage](img/s3/07_iam_start.png?raw=true) - -In case you already have set-up users with IAM before, you will see a list of -them here. Use the "Add user" button at the top to create a new user: -![IAM User List](img/s3/08_user_list.png?raw=true) - -For this tutorial, the new user will be named `restic-demo-user`. Feel free to -choose your own name that best fits your needs. This user will only ever access -AWS through the `restic` program and not through the web interface. Therefore, -"Programmatic access" is selected for "Access type": -![Choosing Username And Access Type](img/s3/09_user_name.png?raw=true) - -During the next step, permissions can be assigned to the new user. To use this -user with Restic, it only needs access to the `restic-demo` bucket. Select -"Attach exiting policies directly", which will bring up a list of pre-defined -policies below. Afterwards, click the "Create policy" button to create a custom -policy: -![Assigning a Policy](img/s3/10_user_pre_policy.png?raw=true) - -A new browser window or tab will open with the policy wizard. In Amazon IAM, -policies are defined as JSON documents. For this tutorial, the "Policy -Generator" will be used to generate a policy file using a web interface: -![Creating a New Policy](img/s3/11_policy_start.png?raw=true) - -After invoking the policy generator, you will be presented with a user -interface to generate individual permission statements. For Restic to work, two -such statements must be created. The first statement is set up as follows: - -``` -Effect: Allow -Service: S3 -Actions: DeleteObject, GetObject, PutObject -Resource: arn:aws:s3:::restic-demo/* -``` - -This statement allows Restic to create, read and delete objects inside the S3 -bucket named `restic-demo`. Adjust the bucket's name to the name of the bucket -you created earlier. Using the "Add Statement" button, this statement can be -saved. Now a second statement is created: - -``` -Effect: Allow -Service: S3 -Actions: ListBucket -Resource: arn:aws:s3:::restic-demo -``` - -Again, substitute `restic-demo` with the actual name of your bucket. Note that, -unlike before, there is no `/*` after the bucket name. This statement allows -Restic to list the objects stored in the `restic-demo` bucket. Again, use "Add -Statement" to save this statement. The policy creator interface should now -look as follows: -![Policy Creator With Two Statements](img/s3/12_policy_permissions_done.png?raw=true) - -Continue to the next step and enter a name and description for this policy. For -this tutorial, the policy will be named `restic-demo-policy`. In this step you -can also examine the JSON document created by the policy generator. Click -"Create Policy" to finish the process: -![Policy Review](img/s3/13_policy_review.png?raw=true) - -Go back to the browser window or tab where you were previously creating the new -user. Click the button labeled "Refresh" above the list of policies to make -sure the newly created policy is available to you. Afterwards, use the search -function to search for the `restic-demo-policy`. Select this policy using the -checkbox on the left. Then, continue to the next step. -![Attaching Policy To User](img/s3/14_user_attach_policy.png?raw=true) - -The next page will present an overview of the user account that is about to be -created. If everything looks good, click "Create user" to complete the process: -![User Creation Review](img/s3/15_user_review.png?raw=true) - -After the user has been created, its access credentials will be displayed. They -consist of the "Access key ID" (think username), and the "Secret access key" -(think password). Copy these down to a safe place. -![User Credentials](img/s3/16_user_created.png?raw=true) - -You have now completed the configuration in AWS. Feel free to close your web -browser now. - - -## Initializing the Restic repository - -Open a terminal and make sure you have the `restic` binary ready. First, choose -a password to encrypt your backups with. In this tutorial, `apg` is used for -this purpose: - -```console -$ apg -a 1 -m 32 -n 1 -M NCL -I9n7G7G0ZpDWA3GOcJbIuwQCGvGUBkU5 -``` - -Note this password somewhere safe along with your AWS credentials. Next, the -configuration of Restic will be placed into environment variables. This will -include sensitive information, such as your AWS secret and repository password. -Therefore, make sure the next commands **do not** end up in your shell's -history file. Adjust the contents of the environment variables to fit your -bucket's name and your user's API credentials. - -```console -$ unset HISTFILE -$ export RESTIC_REPOSITORY="s3:https://s3.amazonaws.com/restic-demo" -$ export AWS_ACCESS_KEY_ID="AKIAJAJSLTZCAZ4SRI5Q" -$ export AWS_SECRET_ACCESS_KEY="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" -$ export RESTIC_PASSWORD="I9n7G7G0ZpDWA3GOcJbIuwQCGvGUBkU5" -``` - -After the environment is set up, Restic may be called to initialize the -repository: - - -```console -$ ./restic init -created restic backend b5c661a86a at s3:https://s3.amazonaws.com/restic-demo - -Please note that knowledge of your password is required to access -the repository. Losing your password means that your data is -irrecoverably lost. -``` - -Restic is now ready to be used with AWS S3. Try to create a backup: - -```console -$ dd if=/dev/urandom bs=1M count=10 of=test.bin -10+0 records in -10+0 records out -10485760 bytes (10 MB, 10 MiB) copied, 0,0891322 s, 118 MB/s - -$ ./restic backup test.bin -scan [/home/philip/restic-demo/test.bin] -scanned 0 directories, 1 files in 0:00 -[0:04] 100.00% 2.500 MiB/s 10.000 MiB / 10.000 MiB 1 / 1 items ... ETA 0:00 -duration: 0:04, 2.47MiB/s -snapshot 10fdbace saved - -$ ./restic snapshots -ID Date Host Tags Directory ----------------------------------------------------------------------- -10fdbace 2017-03-26 16:41:50 blackbox /home/philip/restic-demo/test.bin -``` - -A snapshot was created and stored in the S3 bucket. This snapshot may now be -restored: - -```console -$ mkdir restore - -$ ./restic restore 10fdbace --target restore -restoring to restore - -$ ls restore/ -test.bin -``` - -The snapshot was successfully restored. This concludes the tutorial. - diff --git a/doc/img/s3/01_aws_start.png b/doc/images/aws_s3/01_aws_start.png similarity index 100% rename from doc/img/s3/01_aws_start.png rename to doc/images/aws_s3/01_aws_start.png diff --git a/doc/img/s3/02_aws_menu.png b/doc/images/aws_s3/02_aws_menu.png similarity index 100% rename from doc/img/s3/02_aws_menu.png rename to doc/images/aws_s3/02_aws_menu.png diff --git a/doc/img/s3/03_buckets_list_before.png b/doc/images/aws_s3/03_buckets_list_before.png similarity index 100% rename from doc/img/s3/03_buckets_list_before.png rename to doc/images/aws_s3/03_buckets_list_before.png diff --git a/doc/img/s3/04_bucket_create_start.png b/doc/images/aws_s3/04_bucket_create_start.png similarity index 100% rename from doc/img/s3/04_bucket_create_start.png rename to doc/images/aws_s3/04_bucket_create_start.png diff --git a/doc/img/s3/05_bucket_create_review.png b/doc/images/aws_s3/05_bucket_create_review.png similarity index 100% rename from doc/img/s3/05_bucket_create_review.png rename to doc/images/aws_s3/05_bucket_create_review.png diff --git a/doc/img/s3/06_buckets_list_after.png b/doc/images/aws_s3/06_buckets_list_after.png similarity index 100% rename from doc/img/s3/06_buckets_list_after.png rename to doc/images/aws_s3/06_buckets_list_after.png diff --git a/doc/img/s3/07_iam_start.png b/doc/images/aws_s3/07_iam_start.png similarity index 100% rename from doc/img/s3/07_iam_start.png rename to doc/images/aws_s3/07_iam_start.png diff --git a/doc/img/s3/08_user_list.png b/doc/images/aws_s3/08_user_list.png similarity index 100% rename from doc/img/s3/08_user_list.png rename to doc/images/aws_s3/08_user_list.png diff --git a/doc/img/s3/09_user_name.png b/doc/images/aws_s3/09_user_name.png similarity index 100% rename from doc/img/s3/09_user_name.png rename to doc/images/aws_s3/09_user_name.png diff --git a/doc/img/s3/10_user_pre_policy.png b/doc/images/aws_s3/10_user_pre_policy.png similarity index 100% rename from doc/img/s3/10_user_pre_policy.png rename to doc/images/aws_s3/10_user_pre_policy.png diff --git a/doc/img/s3/11_policy_start.png b/doc/images/aws_s3/11_policy_start.png similarity index 100% rename from doc/img/s3/11_policy_start.png rename to doc/images/aws_s3/11_policy_start.png diff --git a/doc/img/s3/12_policy_permissions_done.png b/doc/images/aws_s3/12_policy_permissions_done.png similarity index 100% rename from doc/img/s3/12_policy_permissions_done.png rename to doc/images/aws_s3/12_policy_permissions_done.png diff --git a/doc/img/s3/13_policy_review.png b/doc/images/aws_s3/13_policy_review.png similarity index 100% rename from doc/img/s3/13_policy_review.png rename to doc/images/aws_s3/13_policy_review.png diff --git a/doc/img/s3/14_user_attach_policy.png b/doc/images/aws_s3/14_user_attach_policy.png similarity index 100% rename from doc/img/s3/14_user_attach_policy.png rename to doc/images/aws_s3/14_user_attach_policy.png diff --git a/doc/img/s3/15_user_review.png b/doc/images/aws_s3/15_user_review.png similarity index 100% rename from doc/img/s3/15_user_review.png rename to doc/images/aws_s3/15_user_review.png diff --git a/doc/img/s3/16_user_created.png b/doc/images/aws_s3/16_user_created.png similarity index 100% rename from doc/img/s3/16_user_created.png rename to doc/images/aws_s3/16_user_created.png diff --git a/doc/index.rst b/doc/index.rst index b2ec658bc..6904005f3 100644 --- a/doc/index.rst +++ b/doc/index.rst @@ -7,6 +7,7 @@ Restic Documentation installation manual faq + tutorials development references talks diff --git a/doc/tutorial_aws_s3.rst b/doc/tutorial_aws_s3.rst new file mode 100644 index 000000000..6fe728418 --- /dev/null +++ b/doc/tutorial_aws_s3.rst @@ -0,0 +1,255 @@ +Setting up Restic with Amazon S3 +================================ + +Preface +------- + +This tutorial will show you how to use Restic with AWS S3. It will show you how +to navigate the AWS web interface, create an S3 bucket, create a user with +access to only this bucket, and finally how to connect Restic to this bucket. + +Prerequisites +------------- + +You should already have a ``restic`` binary available on your system that you can +run. Furthermore, you should also have an account with +`AWS `__. You will likely need to provide credit card +details for billing purposes, even if you use their +`free-tier `__. + + +Logging into AWS +---------------- + +Point your browser to +https://console.aws.amazon.com +and log in using your AWS account. You will be presented with the AWS homepage: + +.. image:: images/aws_s3/01_aws_start.png + :alt: AWS Homepage + +By using the "Services" button in the upper left corder, a menu of all services +provided by AWS can be opened: + +.. image:: images/aws_s3/02_aws_menu.png + :alt: AWS Services Menu + +For this tutorial, the Simple Storage Service (S3), as well as Identity and +Access Management (IAM) are relevant. + + +Creating the bucket +------------------- + +First, a bucket to store your backups in must be created. Using the "Services" +menu, navigate to S3. In case you already have some S3 buckets, you will see a +list of them here: + +.. image:: images/aws_s3/03_buckets_list_before.png + :alt: List of S3 Buckets + +Click the "Create bucket" button and choose a name and region for your new +bucket. For the purpose of this tutorial, the bucket will be named +``restic-demo`` and reside in Frankfurk. Because the bucket name space is +shared among all AWS users, the name ``restic-demo`` may not be available to +you. Be creative and choose a unique bucket name. + +.. image:: images/aws_s3/04_bucket_create_start.png + :alt: Create a Bucket + +It is not necessary to configure any special properties or permissions of the +bucket just yet. Therefore, just finish the wizard without making any further +changes: + +.. image:: images/aws_s3/05_bucket_create_review.png + :alt: Review Bucket Creation + +The newly created ``restic-demo`` bucket will no appear on the list of S3 +buckets: + +.. image:: images/aws_s3/06_buckets_list_after.png + :alt: List With New Bucket + +Creating a user +--------------- + +Use the "Services" menu of the AWS web interface to navigate to IAM. This will +bring you to the IAM homepage. To create a new user, click on the "Users" menu +entry on the left: + +.. image:: images/aws_s3/07_iam_start.png + :alt: IAM Home Page + +In case you already have set-up users with IAM before, you will see a list of +them here. Use the "Add user" button at the top to create a new user: + +.. image:: images/aws_s3/08_user_list.png + :alt: IAM User List + +For this tutorial, the new user will be named ``restic-demo-user``. Feel free to +choose your own name that best fits your needs. This user will only ever access +AWS through the ``restic`` program and not through the web interface. Therefore, +"Programmatic access" is selected for "Access type": + +.. image:: images/aws_s3/09_user_name.png + :alt: Choose User Name and Access Type + +During the next step, permissions can be assigned to the new user. To use this +user with Restic, it only needs access to the ``restic-demo`` bucket. Select +"Attach exiting policies directly", which will bring up a list of pre-defined +policies below. Afterwards, click the "Create policy" button to create a custom +policy: + +.. image:: images/aws_s3/10_user_pre_policy.png + :alt: Assign a Policy + +A new browser window or tab will open with the policy wizard. In Amazon IAM, +policies are defined as JSON documents. For this tutorial, the "Policy +Generator" will be used to generate a policy file using a web interface: + +.. image:: images/aws_s3/11_policy_start.png + :alt: Create a New Policy + +After invoking the policy generator, you will be presented with a user +interface to generate individual permission statements. For Restic to work, two +such statements must be created. The first statement is set up as follows: + +.. code:: + + Effect: Allow + Service: S3 + Actions: DeleteObject, GetObject, PutObject + Resource: arn:aws:s3:::restic-demo/* + +This statement allows Restic to create, read and delete objects inside the S3 +bucket named ``restic-demo``. Adjust the bucket's name to the name of the bucket +you created earlier. Using the "Add Statement" button, this statement can be +saved. Now a second statement is created: + +.. code:: + + Effect: Allow + Service: S3 + Actions: ListBucket + Resource: arn:aws:s3:::restic-demo + +Again, substitute ``restic-demo`` with the actual name of your bucket. Note that, +unlike before, there is no ``/*`` after the bucket name. This statement allows +Restic to list the objects stored in the ``restic-demo`` bucket. Again, use "Add +Statement" to save this statement. The policy creator interface should now +look as follows: + +.. image:: images/aws_s3/12_policy_permissions_done.png + :alt: Policy Creator With Two Statements + +Continue to the next step and enter a name and description for this policy. For +this tutorial, the policy will be named ``restic-demo-policy``. In this step you +can also examine the JSON document created by the policy generator. Click +"Create Policy" to finish the process: + +.. image:: images/aws_s3/13_policy_review.png + :alt: Policy Review + +Go back to the browser window or tab where you were previously creating the new +user. Click the button labeled "Refresh" above the list of policies to make +sure the newly created policy is available to you. Afterwards, use the search +function to search for the ``restic-demo-policy``. Select this policy using the +checkbox on the left. Then, continue to the next step. + +.. image:: images/aws_s3/14_user_attach_policy.png + :alt: Attach Policy to User + +The next page will present an overview of the user account that is about to be +created. If everything looks good, click "Create user" to complete the process: + +.. image:: images/aws_s3/15_user_review.png + :alt: User Creation Review + +After the user has been created, its access credentials will be displayed. They +consist of the "Access key ID" (think user name), and the "Secret access key" +(think password). Copy these down to a safe place. + +.. image:: images/aws_s3/16_user_created.png + :alt: User Credentials + +You have now completed the configuration in AWS. Feel free to close your web +browser now. + + +Initializing the Restic repository +---------------------------------- + +Open a terminal and make sure you have the ``restic`` binary ready. First, choose +a password to encrypt your backups with. In this tutorial, ``apg`` is used for +this purpose: + +.. code-block:: console + + $ apg -a 1 -m 32 -n 1 -M NCL + I9n7G7G0ZpDWA3GOcJbIuwQCGvGUBkU5 + +Note this password somewhere safe along with your AWS credentials. Next, the +configuration of Restic will be placed into environment variables. This will +include sensitive information, such as your AWS secret and repository password. +Therefore, make sure the next commands **do not** end up in your shell's +history file. Adjust the contents of the environment variables to fit your +bucket's name and your user's API credentials. + +.. code-block:: console + + $ unset HISTFILE + $ export RESTIC_REPOSITORY="s3:https://s3.amazonaws.com/restic-demo" + $ export AWS_ACCESS_KEY_ID="AKIAJAJSLTZCAZ4SRI5Q" + $ export AWS_SECRET_ACCESS_KEY="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" + $ export RESTIC_PASSWORD="I9n7G7G0ZpDWA3GOcJbIuwQCGvGUBkU5" + + +After the environment is set up, Restic may be called to initialize the +repository: + + +.. code-block:: console + + $ ./restic init + created restic backend b5c661a86a at s3:https://s3.amazonaws.com/restic-demo + + Please note that knowledge of your password is required to access + the repository. Losing your password means that your data is + irrecoverably lost. + +Restic is now ready to be used with AWS S3. Try to create a backup: + +.. code-block:: console + + $ dd if=/dev/urandom bs=1M count=10 of=test.bin + 10+0 records in + 10+0 records out + 10485760 bytes (10 MB, 10 MiB) copied, 0,0891322 s, 118 MB/s + + $ ./restic backup test.bin + scan [/home/philip/restic-demo/test.bin] + scanned 0 directories, 1 files in 0:00 + [0:04] 100.00% 2.500 MiB/s 10.000 MiB / 10.000 MiB 1 / 1 items ... ETA 0:00 + duration: 0:04, 2.47MiB/s + snapshot 10fdbace saved + + $ ./restic snapshots + ID Date Host Tags Directory + ---------------------------------------------------------------------- + 10fdbace 2017-03-26 16:41:50 blackbox /home/philip/restic-demo/test.bin + +A snapshot was created and stored in the S3 bucket. This snapshot may now be +restored: + +.. code-block:: console + + $ mkdir restore + + $ ./restic restore 10fdbace --target restore + restoring to restore + + $ ls restore/ + test.bin + +The snapshot was successfully restored. This concludes the tutorial. + diff --git a/doc/tutorials.rst b/doc/tutorials.rst new file mode 100644 index 000000000..60632d088 --- /dev/null +++ b/doc/tutorials.rst @@ -0,0 +1,5 @@ +========== +Tutorials +========== + +.. include:: tutorial_aws_s3.rst