christianlight/tutor
6
0
Fork 0
mirror of https://github.com/ChristianLight/tutor.git synced 2024-12-12 06:07:56 +00:00
Code Issues Releases Activity

Easily configurable web proxy

Browse Source 
A web proxy is a web server on the host that will forward request to the
tutor web server. We support Apache2 and Nginx.
This commit is contained in:
Régis Behmo 2019-03-21 17:38:01 +01:00
parent 566fca3c34
commit f45a24caea
13 changed files with 183 additions and 54 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
CHANGELOG.md
View File

@ -2,6 +2,7 @@
## Latest
- [Feature] Easily configure web proxy on the host
- [Bugfix] Fix `images pull all` command which failed on "all" image
- [Improvement] Add configurable mongodb, SMTP and rabbitmq authentication
- [Improvement] Harmonize mysql username/password configuration parameters

78
docs/local.rst
View File

@ -7,8 +7,15 @@ This method is for deploying Open edX locally on a single server, where docker i
In the following, environment and data files will be generated in a user-specific project folder which will be referred to as the "**project root**". On Linux, the default project root is ``~/.local/share/tutor``. An alternative project root can be defined by passing the ``--root=...`` option to most commands, or define the ``TUTOR_ROOT=...`` environment variable.
Main commands
-------------
All available commands can be listed by running::
tutor local help
All-in-one command
------------------
~~~~~~~~~~~~~~~~~~
A fully-functional platform can be configured and run in one command::
@ -17,7 +24,7 @@ A fully-functional platform can be configured and run in one command::
But you may want to run commands one at a time: it's faster when you need to run only part of the local deployment process, and it helps you understand how your platform works. In the following we decompose the ``quickstart`` command.
Configuration
-------------
~~~~~~~~~~~~~
::
@ -28,7 +35,7 @@ This is the only non-automatic step in the install process. You will be asked va
If you want to run a fully automated install, upload the ``config.yml`` file to wherever you want to run Open edX. You can then entirely skip the configuration step.
Update docker images
--------------------
~~~~~~~~~~~~~~~~~~~~
::
@ -37,7 +44,7 @@ Update docker images
This downloads the latest version of the docker images from `Docker Hub <https://hub.docker.com/r/regis/openedx/>`_. Depending on your bandwidth, this might take a long time. Minor image updates will be incremental, and thus much faster.
Database management
-------------------
~~~~~~~~~~~~~~~~~~~
::
@ -48,7 +55,7 @@ This command should be run just once. It will create the required databases tabl
If migrations are stopped with a ``Killed`` message, this certainly means the docker containers don't have enough RAM. See the :ref:`troubleshooting` section.
Running Open edX
----------------
~~~~~~~~~~~~~~~~
::
@ -66,8 +73,11 @@ And then, to stop all services::
tutor local stop
Extra commands
--------------
Creating a new user with staff and admin rights
-----------------------------------------------
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You will most certainly need to create a user to administer the platform. Just run::
@ -76,14 +86,14 @@ You will most certainly need to create a user to administer the platform. Just r
You will asked to set the user password interactively.
Importing the demo course
-------------------------
~~~~~~~~~~~~~~~~~~~~~~~~~
On a fresh install, your platform will not have a single course. To import the `Open edX demo course <https://github.com/edx/edx-demo-course>`_, run::
tutor local importdemocourse
Updating the course search index
--------------------------------
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The course search index can be updated with::
@ -94,7 +104,7 @@ Run this command periodically to ensure that course search results are always up
.. _portainer:
Docker container web UI with `Portainer <https://portainer.io/>`__
------------------------------------------------------------------
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Portainer is a web UI for managing docker containers. It lets you view your entire Open edX platform at a glace. Try it! It's really cool::
@ -107,25 +117,49 @@ After launching your platfom, the web UI will be available at `http://localhost:
Among many other things, you'll be able to view the logs for each container, which is really useful.
Recipes
-------
.. _web_proxy:
Running Open edX behind a web proxy
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The containerized web server (nginx) needs to listen to ports 80 and 443 on the host. If there is already a webserver running on the host, such as Apache or Nginx, the nginx container will not be able to start. Tutor supports running behind a web proxy. To do so, add the following configuration::
tutor config save --silent --set WEB_PROXY=true --set NGINX_HTTP_PORT=81 --set NGINX_HTTPS_PORT=444
In this example, the nginx container ports would be mapped to 81 and 444, instead of 80 and 443. You must then configure the web proxy on the host. Basic configuration files are provided by Tutor which can be used directly by your web proxy.
For nginx::
sudo ln -s $(tutor config printroot)/env/local/proxy/nginx/openedx.conf /etc/nginx/sites-enabled/
sudo systemctl reload nginx
For apache::
sudo a2enmod proxy
sudo a2enmod proxy_http
sudo ln -s $(tutor config printroot)/env/local/proxy/apache2/openedx.conf /etc/apache2/sites-enabled/
sudo systemctl reload apache2
If you have configured your platform to use SSL/TLS certificates for HTTPS access, the generation and renewal of certificates will not be managed by Tutor: you are supposed to take care of it yourself. Suggestions for generating and renewing these certificates with `Let's Encrypt <https://letsencrypt.org/>`_ are given by::
tutor local https create
tutor local https renew
Loading different production settings for ``edx-platform``
----------------------------------------------------------
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The default settings module loaded by ``edx-platform`` is ``tutor.production``. The folders ``tutor/deploy/env/openedx/settings/lms`` and ``tutor/deploy/env/openedx/settings/cms`` are mounted as ``edx-platform/lms/envs/tutor`` and ``edx-platform/cms/envs/tutor`` inside the docker containers. Thus, to use your own settings, you must do two things:
The default settings module loaded by ``edx-platform`` is ``tutor.production``. The folders ``$(tutor config printroot)/env/apps/openedx/settings/lms`` and ``$(tutor config printroot)/env/apps/openedx/settings/cms`` are mounted as ``edx-platform/lms/envs/tutor`` and ``edx-platform/cms/envs/tutor`` inside the docker containers. Thus, to use your own settings, you must do two things:
1. Copy your settings files for the lms and the cms to ``tutor/deploy/env/openedx/settings/lms/mysettings.py`` and ``tutor/deploy/env/openedx/settings/cms/mysettings.py``.
2. Load your settings by adding ``EDX_PLATFORM_SETTINGS=tutor.mysettings`` to ``tutor/deploy/local/.env``.
1. Copy your settings files for the lms and the cms to ``$(tutor config printroot)/env/apps/openedx/settings/lms/mysettings.py`` and ``$(tutor config printroot)/env/apps/openedx/settings/cms/mysettings.py``.
2. Load your settings by adding ``EDX_PLATFORM_SETTINGS=tutor.mysettings`` to ``$(tutor config printroot)/env/local/.env``.
Of course, your settings should be compatible with the docker install. You can get some inspiration from the ``production.py`` settings modules generated by Tutor.
Additional commands
-------------------
All available commands can be listed by running::
tutor local help
Of course, your settings should be compatible with the docker install. You can get some inspiration from the ``production.py`` settings modules generated by Tutor, or just import it as a base by adding ``from .production import *`` at the top of ``mysettings.py``.
Upgrading from earlier versions
-------------------------------
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Versions 1 and 2 of Tutor were organized differently: they relied on many different ``Makefile`` and ``make`` commands instead of a single ``tutor`` executable. To migrate from an earlier version, you should first stop your platform::

17
docs/troubleshooting.rst
View File

@ -34,22 +34,7 @@ If you'd rather use a graphical user interface for viewing logs, you are encoura
"Cannot start service nginx: driver failed programming external connectivity"
-----------------------------------------------------------------------------
The containerized Nginx needs to listen to ports 80 and 443 on the host. If there is already a webserver, such as Apache or Nginx, running on the host, the nginx container will not be able to start. There are two solutions:
1. Stop Apache or Nginx on the host::
sudo systemctl stop apache2
sudo systemctl stop nginx
However, you might not want to do that if you need a webserver for running non-Open edX related applications. In such cases...
2. Run the nginx container on different ports: to do so, configure different ports for the dockerized ngins::
tutor config save --silent --set NGINX_HTTP_PORT=81 --set NGINX_HTTPS_PORT=444
In this example, the nginx container ports would be mapped to 81 and 444, instead of 80 and 443. Then, restart nginx with ``tutor local restart nginx``.
You should note that with the latter solution, it is your responsibility to configure the webserver on the host as a proxy to the nginx container. See `this github issue <https://github.com/regisb/tutor/issues/69#issuecomment-425916825>`_ for http, and `this other github issue <https://github.com/regisb/tutor/issues/90#issuecomment-437687294>`_ for https.
The containerized Nginx needs to listen to ports 80 and 443 on the host. If there is already a webserver, such as Apache or Nginx, running on the host, the nginx container will not be able to start. To solve this issue, check :ref:`web_proxy`.
Help! The Docker containers are eating all my RAM/CPU/CHEESE
------------------------------------------------------------

6
tutor/env.py
View File

@ -30,7 +30,11 @@ def render_target(root, config, target):
for src, dst in walk_templates(root, target):
if is_part_of_env(src):
with codecs.open(src, encoding='utf-8') as fi:
substituted = render_str(fi.read(), config)
try:
substituted = render_str(fi.read(), config)
except jinja2.exceptions.TemplateError:
print("Error rendering template", src)
raise
with open(dst, "w") as of:
of.write(substituted)

26
tutor/local.py
View File

@ -1,10 +1,12 @@
import os
import subprocess
from textwrap import indent
from time import sleep
import click
from . import config as tutor_config
from . import exceptions
from . import fmt
from . import opts
from . import scripts
@ -56,6 +58,7 @@ def start(root, detach):
command = ["up", "--remove-orphans"]
if detach:
command.append("-d")
docker_compose(root, *command)
if detach:
@ -63,7 +66,7 @@ def start(root, detach):
config = tutor_config.load(root)
http = "https" if config["ACTIVATE_HTTPS"] else "http"
urls = []
if not config["ACTIVATE_HTTPS"]:
if not config["ACTIVATE_HTTPS"] and not config["WEB_PROXY"]:
urls += [
"http://localhost",
"http://studio.localhost",
@ -163,12 +166,23 @@ def https_create(root):
click.echo(fmt.info("HTTPS is not activated: certificate generation skipped"))
return
script = tutor_env.render_str(scripts.https_certificates_create, config)
if config['WEB_PROXY']:
click.echo(fmt.info(
"""You are running Tutor behind a web proxy (WEB_PROXY=true): SSL/TLS certificates must be generated on the host. For instance, to generate certificates with Let's Encrypt, run:
{}
See the official certbot documentation for your platform: https://certbot.eff.org/""".format(indent(script, " "))))
return
utils.docker_run(
"--volume", "{}:/etc/letsencrypt/".format(tutor_env.data_path(root, "letsencrypt")),
"-p", "80:80",
"--entrypoint=sh",
"certbot/certbot:latest",
"-c", tutor_env.render_str(scripts.https_certificates_create, config),
"-c", script,
)
@click.command(help="Renew https certificates", name="renew")
@ -178,6 +192,14 @@ def https_renew(root):
if not config['ACTIVATE_HTTPS']:
click.echo(fmt.info("HTTPS is not activated: certificate renewal skipped"))
return
if config['WEB_PROXY']:
click.echo(fmt.info(
"""You are running Tutor behind a web proxy (WEB_PROXY=true): SSL/TLS certificates must be renewed on the host. For instance, to renew Let's Encrypt certificates, run:
certbot renew
See the official certbot documentation: for your platform https://certbot.eff.org/"""))
return
docker_run = [
"--volume", "{}:/etc/letsencrypt/".format(tutor_env.data_path(root, "letsencrypt")),
"-p", "80:80",

4
tutor/scripts.py
View File

@ -34,9 +34,7 @@ oauth2 = """
{% endif %}"""
https_certificates_create = """certbot certonly --standalone -n --agree-tos -m admin@{{ LMS_HOST }} -d {{ LMS_HOST }} -d {{ CMS_HOST }} -d preview.{{ LMS_HOST }}
{% if ACTIVATE_NOTES %}
certbot certonly --standalone -n --agree-tos -m admin@{{ LMS_HOST }} -d notes.{{ LMS_HOST }}
{% endif %}"""
{% if ACTIVATE_NOTES %}certbot certonly --standalone -n --agree-tos -m admin@{{ LMS_HOST }} -d notes.{{ LMS_HOST }}{% endif %}"""
create_user = """./manage.py lms --settings=tutor.production manage_user {{ OPTS }} {{ USERNAME }} {{ EMAIL }}
./manage.py lms --settings=tutor.production changepassword {{ USERNAME }}"""

6
tutor/templates/apps/nginx/cms.conf
View File

@ -2,7 +2,7 @@ upstream cms-backend {
server cms:8000 fail_timeout=0;
}
{% if ACTIVATE_HTTPS %}
{% if ACTIVATE_HTTPS and not WEB_PROXY %}
server {
server_name {{ CMS_HOST }};
listen 80;
@ -11,10 +11,10 @@ server {
{% endif %}
server {
listen {{ "443 ssl" if ACTIVATE_HTTPS else "80" }};
listen {{ "443 ssl" if ACTIVATE_HTTPS and not WEB_PROXY else "80" }};
server_name studio.localhost {{ CMS_HOST }};
{% if ACTIVATE_HTTPS %}
{% if ACTIVATE_HTTPS and not WEB_PROXY %}
ssl_certificate /etc/letsencrypt/live/{{ LMS_HOST }}/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/{{ LMS_HOST }}/privkey.pem;
{% endif %}

6
tutor/templates/apps/nginx/extra.conf
View File

@ -3,7 +3,7 @@ upstream notes-backend {
server notes:8000 fail_timeout=0;
}
{% if ACTIVATE_HTTPS %}
{% if ACTIVATE_HTTPS and not WEB_PROXY %}
server {
server_name notes.{{ LMS_HOST }};
listen 80;
@ -12,10 +12,10 @@ server {
{% endif %}
server {
listen {{ "443 ssl" if ACTIVATE_HTTPS else "80" }};
listen {{ "443 ssl" if ACTIVATE_HTTPS and not WEB_PROXY else "80" }};
server_name notes.localhost notes.{{ LMS_HOST }};
{% if ACTIVATE_HTTPS %}
{% if ACTIVATE_HTTPS and not WEB_PROXY %}
ssl_certificate /etc/letsencrypt/live/notes.{{ LMS_HOST }}/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/notes.{{ LMS_HOST }}/privkey.pem;
{% endif %}

6
tutor/templates/apps/nginx/lms.conf
View File

@ -2,7 +2,7 @@ upstream lms-backend {
server lms:8000 fail_timeout=0;
}
{% if ACTIVATE_HTTPS %}
{% if ACTIVATE_HTTPS and not WEB_PROXY %}
server {
server_name {{ LMS_HOST }} preview.{{ LMS_HOST }};
listen 80;
@ -11,10 +11,10 @@ server {
{% endif %}
server {
listen {{ "443 ssl" if ACTIVATE_HTTPS else "80" }};
listen {{ "443 ssl" if ACTIVATE_HTTPS and not WEB_PROXY else "80" }};
server_name localhost preview.localhost {{ LMS_HOST }} preview.{{ LMS_HOST }};
{% if ACTIVATE_HTTPS %}
{% if ACTIVATE_HTTPS and not WEB_PROXY %}
ssl_certificate /etc/letsencrypt/live/{{ LMS_HOST }}/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/{{ LMS_HOST }}/privkey.pem;
{% endif %}

1
tutor/templates/config-defaults.yml
View File

@ -45,6 +45,7 @@ SMTP_HOST: "smtp"
SMTP_PORT: 25
SMTP_USERNAME: ""
SMTP_PASSWORD: ""
WEB_PROXY: false
XQUEUE_AUTH_USERNAME: "lms"
XQUEUE_MYSQL_DATABASE: "xqueue"
XQUEUE_MYSQL_USERNAME: "xqueue"

35
tutor/templates/local/proxy/apache2/openedx.conf Normal file
View File

@ -0,0 +1,35 @@
<VirtualHost *:80>
ServerName {{ LMS_HOST }}
ServerAlias preview.{{ LMS_HOST }} {{ CMS_HOST }} {% if ACTIVATE_NOTES %}notes.{{ LMS_HOST }}{% endif %}
ProxyPreserveHost On
ProxyRequests On
ProxyPass / http://localhost:{{ NGINX_HTTP_PORT }}/
ProxyPassReverse / http://localhost:{{ NGINX_HTTP_PORT }}/
</VirtualHost>
{% if ACTIVATE_HTTPS %}
<VirtualHost *:443>
ServerName {{ LMS_HOST }}
ServerAlias preview.{{ LMS_HOST }} {{ CMS_HOST }}
SSLEngine on
SSLCertificateFile /etc/letsencrypt/live/{{ LMS_HOST }}/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/{{ LMS_HOST }}/privkey.pem
ProxyPreserveHost On
ProxyRequests On
ProxyPass / http://localhost:{{ NGINX_HTTPS_PORT }}/
ProxyPassReverse / http://localhost:{{ NGINX_HTTPS_PORT }}/
</VirtualHost>
<VirtualHost *:443>
ServerName notes.{{ LMS_HOST }}
SSLEngine on
SSLCertificateFile /etc/letsencrypt/live/notes.{{ LMS_HOST }}/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/notes.{{ LMS_HOST }}/privkey.pem
ProxyPreserveHost On
ProxyRequests On
ProxyPass / http://localhost:{{ NGINX_HTTPS_PORT }}/
ProxyPassReverse / http://localhost:{{ NGINX_HTTPS_PORT }}/
</VirtualHost>
{% endif %}

49
tutor/templates/local/proxy/nginx/openedx.conf Normal file
View File

@ -0,0 +1,49 @@
{% if ACTIVATE_HTTPS %}
server {
server_name {{ LMS_HOST }} preview.{{ LMS_HOST }} {{ CMS_HOST }}{% if ACTIVATE_NOTES %} notes.{{ LMS_HOST }}{% endif %};
listen 80;
return 301 https://$server_name$request_uri;
}
{% endif %}
server {
listen {{ "443 ssl" if ACTIVATE_HTTPS else "80" }};
server_name {{ LMS_HOST }} preview.{{ LMS_HOST }} {{ CMS_HOST }};
{% if ACTIVATE_HTTPS %}
ssl_certificate /etc/letsencrypt/live/{{ LMS_HOST }}/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/{{ LMS_HOST }}/privkey.pem;
{% endif %}
server_tokens off;
location / {
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-Port $server_port;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header Host $http_host;
proxy_redirect off;
proxy_pass http://localhost:{{ NGINX_HTTPS_PORT if ACTIVATE_HTTPS else NGINX_HTTP_PORT }};
}
}
{% if ACTIVATE_NOTES %}
server {
listen {{ "443 ssl" if ACTIVATE_HTTPS else "80" }};
server_name notes.{{ LMS_HOST }};
{% if ACTIVATE_HTTPS %}
ssl_certificate /etc/letsencrypt/live/notes.{{ LMS_HOST }}/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/notes.{{ LMS_HOST }}/privkey.pem;
{% endif %}
server_tokens off;
location / {
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-Port $server_port;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header Host $http_host;
proxy_redirect off;
proxy_pass http://localhost:{{ NGINX_HTTPS_PORT if ACTIVATE_HTTPS else NGINX_HTTP_PORT }};
}
}
{% endif %}

2
tutor/utils.py
View File

@ -50,7 +50,7 @@ def execute(*command):
p.kill()
p.wait()
raise
except Exception as e:
except Exception:
p.kill()
p.wait()
raise exceptions.TutorError("Command failed: {}".format(