feat: get rid of the nginx container and services

Nginx and Caddy performed duplicate tasks. It was decided to get rid of
the nginx container, for simplification. This is a breaking change for
plugin developers. Also, applications that collect nginx logs will have
to be modified.

See:
- Corresponding TEP: https://discuss.overhang.io/t/tep-get-rid-of-the-nginx-container/2024
- the prior discussion: https://discuss.overhang.io/t/why-caddy-nginx/1952

CHANGELOG-nightly.md

@@ -1,3 +1,8 @@
 # Changelog (nightly branch)
 
 Note: Breaking changes between versions are indicated by "💥".
+
+- 💥[Feature] Get rid of the nginx container and service, which is now replaced by Caddy. this has the following consequences:
+    - Patches "nginx-cms", "nginx-lms", "nginx-extra", "local-docker-compose-nginx-aliases" are replaced by "caddyfile-cms", "caddyfile-lms", "caddyfile", " local-docker-compose-caddy-aliases".
+    - Patches "k8s-deployments-nginx-volume-mounts", "k8s-deployments-nginx-volumes" were obsolete and are removed.
+    - The `NGINX_HTTP_PORT` setting is renamed to `CADDY_HTTP_PORT`.

docs/configuration.rst

@@ -99,16 +99,10 @@ Vendor services
 Caddy
 *****
 
-- ``RUN_CADDY`` (default: ``true``)
+- ``CADDY_HTTP_PORT`` (default: ``80``)
+- ``ENABLE_WEB_PROXY`` (default: ``true``)
 
-`Caddy <https://caddyserver.com>`__ is a web server used in Tutor as a web proxy for the generation of SSL/TLS certificates at runtime. If ``RUN_CADDY`` is set to ``false`` then we assume that SSL termination does not occur in the Caddy container, and thus the ``caddy`` container is not started.
-
-Nginx
-*****
-
-- ``NGINX_HTTP_PORT`` (default: ``80``)
-
-Nginx is used to route web traffic to the various applications and to serve static assets. When ``RUN_CADDY`` is false, the ``NGINX_HTTP_PORT`` is exposed on the host.
+`Caddy <https://caddyserver.com>`__ is a web server used in Tutor both as a web proxy and for the generation of SSL/TLS certificates at runtime. Port indicated by ``CADDY_HTTP_PORT`` is exposed on the host, in addition to port 443. If ``ENABLE_WEB_PROXY`` is set to ``false`` then we assume that SSL termination does not occur in the Caddy container and only ``CADDY_HTTP_PORT`` is exposed on the host.
 
 MySQL
 *****
@@ -193,7 +187,7 @@ The following DNS records must exist and point to your server::
 
 Thus, **this feature will (probably) not work in development** because the DNS records will (probably) not point to your development machine.
 
-If you would like to perform SSL/TLS termination with your own custom certificates, you will have to keep ``ENABLE_HTTPS=true`` and turn off the Caddy server with ``RUN_CADDY=false``. See the corresponding :ref:`tutorial <web_proxy>` for more information.
+If you would like to perform SSL/TLS termination with your own custom certificates, you will have to keep ``ENABLE_HTTPS=true`` and turn off the Caddy load balancing with ``ENABLE_WEB_PROXY=false``. See the corresponding :ref:`tutorial <web_proxy>` for more information.
 
 .. _customise:
 

docs/troubleshooting.rst

@@ -33,7 +33,7 @@ To view the logs from all containers use the ``tutor local logs`` command, which
 
 To view the logs from just one container, for instance the web server::
 
-    tutor local logs --follow nginx
+    tutor local logs --follow caddy
 
 The last commands produce the logs since the creation of the containers, which can be a lot. Similar to a ``tail -f``, you can run::
 
@@ -43,10 +43,10 @@ If you'd rather use a graphical user interface for viewing logs, you are encoura
 
 .. _webserver:
 
-"Cannot start service nginx: driver failed programming external connectivity"
+"Cannot start service caddy: driver failed programming external connectivity"
 -----------------------------------------------------------------------------
 
-The containerized Nginx needs to listen to ports 80 and 443 on the host. If there is already a webserver, such as Apache or Nginx, running on the host, the nginx container will not be able to start. To solve this issue, check the section on :ref:`how to setup a web proxy <web_proxy>`.
+The containerized Caddy needs to listen to ports 80 and 443 on the host. If there is already a webserver, such as Apache, Caddy or Nginx, running on the host, the caddy container will not be able to start. To solve this issue, check the section on :ref:`how to setup a web proxy <web_proxy>`.
 
 "Couldn't connect to docker daemon"
 -----------------------------------

docs/tutorials/multiplatforms.rst

@@ -5,7 +5,7 @@ With Tutor, it is easy to run multiple Open edX instances on a single server. To
 
 - ``TUTOR_ROOT``: so that configuration, environment and data are not mixed up between platforms.
 - ``LOCAL_PROJECT_NAME``: the various docker-compose projects cannot share the same name.
-- ``NGINX_HTTP_PORT``: ports cannot be shared by two different containers.
+- ``CADDY_HTTP_PORT``: exposed ports cannot be shared by two different containers.
 - ``LMS_HOST``, ``CMS_HOST``: the different platforms must be accessible from different domain (or subdomain) names.
 
 In addition, a web proxy must be setup on the host, as described :ref:`in the corresponding tutorial <web_proxy>`.

docs/tutorials/proxy.rst

@@ -5,11 +5,11 @@ Running Open edX behind a web proxy
 
 The containerized web server (`Caddy <https://caddyserver.com/>`__) needs to listen to ports 80 and 443 on the host. If there is already a webserver running on the host, such as Apache or Nginx, the caddy container will not be able to start. Tutor supports running behind a web proxy. To do so, add the following configuration::
 
-       tutor config save --set RUN_CADDY=false --set NGINX_HTTP_PORT=81
+       tutor config save --set ENABLE_WEB_PROXY=false --set CADDY_HTTP_PORT=81
 
-In this example, the nginx container port would be mapped to 81 instead of 80. You must then configure the web proxy on the host. As of v11.0.0, configuration files are no longer provided for automatic configuration of your web proxy. Basically, you should setup a reverse proxy to `localhost:NGINX_HTTP_PORT` from the following hosts: LMS_HOST, PREVIEW_LMS_HOST, CMS_HOST, as well as any additional host exposed by your plugins.
+In this example, the caddy container port would be mapped to 81 instead of 80. You must then configure the web proxy on the host. As of v11.0.0, configuration files are no longer provided for automatic configuration of your web proxy. Basically, you should setup a reverse proxy to `localhost:CADDY_HTTP_PORT` from the following hosts: LMS_HOST, PREVIEW_LMS_HOST, CMS_HOST, as well as any additional host exposed by your plugins.
 
 .. warning::
-    In this setup, the Nginx HTTP port will be exposed to the world. Make sure to configure your server firewall to block unwanted connections to your server's ``NGINX_HTTP_PORT``. Alternatively, you can configure the Nginx container to accept only local connections::
+    In this setup, the Caddy HTTP port will be exposed to the world. Make sure to configure your server firewall to block unwanted connections to your server's ``CADDY_HTTP_PORT``. Alternatively, you can configure the Caddy container to accept only local connections::
 
-        tutor config save --set NGINX_HTTP_PORT=127.0.0.1:81
+        tutor config save --set CADDY_HTTP_PORT=127.0.0.1:81

tutor/commands/images.py

@@ -18,7 +18,6 @@ VENDOR_IMAGES = [
     "elasticsearch",
     "mongodb",
     "mysql",
-    "nginx",
     "redis",
     "smtp",
 ]

tutor/commands/k8s.py

@@ -165,14 +165,14 @@ def quickstart(context: click.Context, non_interactive: bool) -> None:
     config = interactive_config.update(
         context.obj.root, interactive=(not non_interactive)
     )
-    if not config["RUN_CADDY"]:
+    if not config["ENABLE_WEB_PROXY"]:
         fmt.echo_alert(
-            "Potentially invalid configuration: RUN_CADDY=false\n"
+            "Potentially invalid configuration: ENABLE_WEB_PROXY=false\n"
             "This setting might have been defined because you previously set WEB_PROXY=true. This is no longer"
             " necessary in order to get Tutor to work on Kubernetes. In Tutor v11+ a Caddy-based load balancer is"
             " provided out of the box to handle SSL/TLS certificate generation at runtime. If you disable this"
             " service, you will have to configure an Ingress resource and a certificate manager yourself to redirect"
-            " traffic to the nginx service. See the Kubernetes section in the Tutor documentation for more"
+            " traffic to the caddy service. See the Kubernetes section in the Tutor documentation for more"
             " information."
         )
     click.echo(fmt.title("Updating the current environment"))

tutor/config.py

@@ -174,6 +174,12 @@ def upgrade_obsolete(config: Config) -> None:
     ]:
         if name in config:
             config[name.replace("ACTIVATE_", "RUN_")] = config.pop(name)
+    # Replace RUN_CADDY by ENABLE_WEB_PROXY
+    if "RUN_CADDY" in config:
+        config["ENABLE_WEB_PROXY"] = config.pop("RUN_CADDY")
+    # Replace RUN_CADDY by ENABLE_WEB_PROXY
+    if "NGINX_HTTP_PORT" in config:
+        config["CADDY_HTTP_PORT"] = config.pop("NGINX_HTTP_PORT")
 
 
 def convert_json2yml(root: str) -> None:

tutor/env.py

@@ -222,11 +222,9 @@ def save(root: str, config: Config) -> None:
 
 
 def upgrade_obsolete(root: str) -> None:
-    # tutor.conf was renamed to _tutor.conf in order to be the first config file loaded
-    # by nginx
-    nginx_tutor_conf = pathjoin(root, "apps", "nginx", "tutor.conf")
-    if os.path.exists(nginx_tutor_conf):
-        os.remove(nginx_tutor_conf)
+    """
+    Add here ad-hoc commands to upgrade the environment.
+    """
 
 
 def save_plugin_templates(

tutor/templates/apps/caddy/Caddyfile

@@ -1,13 +1,69 @@
-{{ LMS_HOST }}{% if not ENABLE_HTTPS %}:80{% endif %} {
-    reverse_proxy nginx:80 {
+# Global configuration
+{
+    {{ patch("caddyfile-global")|indent(4) }}
+}
+
+# proxy directive snippet (with logging) to be used as follows:
+#
+#     import proxy "containername:port"
+(proxy) {
+    log {
+        output stdout
+        format filter {
+            wrap json
+            fields {
+                common_log delete
+                request>headers delete
+                resp_headers delete
+                tls delete
+            }
+        }
+    }
+
+    reverse_proxy {args.0} {
         header_up X-Forwarded-Port {{ 443 if ENABLE_HTTPS else 80 }}
     }
 }
-{{ PREVIEW_LMS_HOST }}{% if not ENABLE_HTTPS %}:80{% endif %} {
-    reverse_proxy nginx:80
+
+{% if ENABLE_HTTPS and ENABLE_WEB_PROXY %}
+{% set port = "" %}
+{# listening to https is disabled and we must only listen to http #}
+{% else %}
+{% set port = ":80" %}
+{% endif %}
+
+{{ LMS_HOST }}{{ port }}, {{ PREVIEW_LMS_HOST }}{{ port }} {
+    @favicon_matcher {
+        path_regexp ^(.*)/favicon.ico$
+    }
+    rewrite @favicon_matcher /static/images/favicon.ico
+
+    # Limit profile image upload size
+    request_body /api/profile_images/*/*/upload {
+        max_size 1MB
+    }
+    request_body {
+        max_size 4MB
+    }
+
+    import proxy "lms:8000"
+
+    {{ patch("caddyfile-lms")|indent(4) }}
 }
-{{ CMS_HOST }}{% if not ENABLE_HTTPS %}:80{% endif %} {
-    reverse_proxy nginx:80
+
+{{ CMS_HOST }}{{ port }} {
+    @favicon_matcher {
+        path_regexp ^(.*)/favicon.ico$
+    }
+    rewrite @favicon_matcher /static/images/favicon.ico
+
+    request_body {
+        max_size 250MB
+    }
+
+    import proxy "cms:8000"
+
+    {{ patch("caddyfile-cms")|indent(4) }}
 }
 
 {{ patch("caddyfile") }}

tutor/templates/apps/nginx/_tutor.conf

@@ -1,10 +0,0 @@
-# Allow long domain names
-server_names_hash_bucket_size 128;
-
-# Set a short ttl for proxies to allow restarts
-resolver 127.0.0.11 [::1]:5353 valid=10s;
-
-# Configure logging to include scheme and server name
-log_format tutor '$remote_addr - $remote_user [$time_local] $scheme://$host "$request" '
-                 '$status $body_bytes_sent "$http_referer" '
-                 '"$http_user_agent" "$http_x_forwarded_for"';

tutor/templates/apps/nginx/cms.conf

@@ -1,28 +0,0 @@
-{% if RUN_CMS %}
-upstream cms-backend {
-    server cms:8000 fail_timeout=0;
-}
-
-server {
-  listen 80;
-  server_name {{ CMS_HOST }};
-
-  access_log /var/log/nginx/access.log tutor;
-  client_max_body_size 250M;
-  server_tokens off;
-
-  rewrite ^(.*)/favicon.ico$ /static/images/favicon.ico last;
-
-  location @proxy_to_cms_app {
-    proxy_redirect off;
-    proxy_set_header Host $http_host;
-    proxy_pass http://cms-backend;
-  }
-
-  location / {
-    try_files $uri @proxy_to_cms_app;
-  }
-
-  {{ patch("nginx-cms")|indent(2) }}
-}
-{% endif %}

tutor/templates/apps/nginx/extra.conf

@@ -1 +0,0 @@
-{{ patch("nginx-extra") }}

tutor/templates/apps/nginx/lms.conf

@@ -1,47 +0,0 @@
-{% if RUN_LMS %}
-upstream lms-backend {
-    server lms:8000 fail_timeout=0;
-}
-
-server {
-  listen 80;
-  server_name {{ LMS_HOST }} {{ PREVIEW_LMS_HOST }};
-
-  access_log /var/log/nginx/access.log tutor;
-  client_max_body_size 4M;
-  server_tokens off;
-
-  rewrite ^(.*)/favicon.ico$ /static/images/favicon.ico last;
-
-  # Allow large cookies
-  proxy_buffer_size 8k;
-
-  location @proxy_to_lms_app {
-    proxy_redirect off;
-    proxy_set_header Host $http_host;
-    proxy_pass http://lms-backend;
-  }
-
-  location / {
-    try_files $uri @proxy_to_lms_app;
-  }
-
-  # /login?next=<any image> can be used by 3rd party sites in <img> tags to
-  # determine whether a user on their site is logged into edX.
-  # The most common image to use is favicon.ico.
-  location /login {
-    if ( $arg_next ~* "favicon.ico" ) {
-      return 403;
-    }
-    try_files $uri @proxy_to_lms_app;
-  }
-
-  # Need a separate location for the image uploads endpoint to limit upload sizes
-  location ~ ^/api/profile_images/[^/]*/[^/]*/upload$ {
-    try_files $uri @proxy_to_lms_app;
-    client_max_body_size 1049576;
-  }
-
-  {{ patch("nginx-lms")|indent(2) }}
-}
-{% endif %}

tutor/templates/config.yml

@@ -9,16 +9,15 @@ ID: "{{ 24|random_string }}"
 LMS_HOST: "www.myopenedx.com"
 
 # The following are default values
-RUN_CADDY: true
 RUN_LMS: true
 RUN_CMS: true
 RUN_FORUM: true
 RUN_ELASTICSEARCH: true
-ENABLE_HTTPS: false
 RUN_MONGODB: true
 RUN_MYSQL: true
 RUN_REDIS: true
 RUN_SMTP: true
+CADDY_HTTP_PORT: 80
 CMS_HOST: "studio.{{ LMS_HOST }}"
 PREVIEW_LMS_HOST: "preview.{{ LMS_HOST }}"
 CONTACT_EMAIL: "contact@{{ LMS_HOST }}"
@@ -29,6 +28,7 @@ DOCKER_REGISTRY: "docker.io/"
 DOCKER_IMAGE_OPENEDX: "{{ DOCKER_REGISTRY }}overhangio/openedx:{{ TUTOR_VERSION }}"
 DOCKER_IMAGE_OPENEDX_DEV: "{{ DOCKER_REGISTRY }}overhangio/openedx-dev:{{ TUTOR_VERSION }}"
 DOCKER_IMAGE_CADDY: "{{ DOCKER_REGISTRY }}caddy:2.3.0"
+DOCKER_IMAGE_ELASTICSEARCH: "{{ DOCKER_REGISTRY }}elasticsearch:7.10.1"
 DOCKER_IMAGE_FORUM: "{{ DOCKER_REGISTRY }}overhangio/openedx-forum:{{ TUTOR_VERSION }}"
 DOCKER_IMAGE_MONGODB: "{{ DOCKER_REGISTRY }}mongo:4.2.17"
 DOCKER_IMAGE_MYSQL: "{{ DOCKER_REGISTRY }}mysql:5.7.35"
@@ -41,6 +41,8 @@ ELASTICSEARCH_HOST: "elasticsearch"
 ELASTICSEARCH_PORT: 9200
 ELASTICSEARCH_SCHEME: "http"
 ELASTICSEARCH_HEAP_SIZE: 1g
+ENABLE_HTTPS: false
+ENABLE_WEB_PROXY: true
 FORUM_HOST: "forum"
 FORUM_MONGODB_DATABASE: "cs_comments_service"
 JWT_COMMON_AUDIENCE: "openedx"
@@ -65,7 +67,6 @@ OPENEDX_COMMON_VERSION: "master"
 MYSQL_HOST: "mysql"
 MYSQL_PORT: 3306
 MYSQL_ROOT_USERNAME: "root"
-NGINX_HTTP_PORT: 80
 PLATFORM_NAME: "My Open edX"
 PLUGINS: []
 PREVIEW_LMS_HOST: "preview.{{ LMS_HOST }}"

tutor/templates/k8s/deployments.yml

@@ -1,4 +1,4 @@
-{% if RUN_CADDY %}
+{% if ENABLE_WEB_PROXY %}
 ---
 apiVersion: apps/v1
 kind: Deployment
@@ -379,36 +379,6 @@ spec:
           ports:
             - containerPort: 25
 {% endif %}
----
-apiVersion: apps/v1
-kind: Deployment
-metadata:
-  name: nginx
-  labels:
-    app.kubernetes.io/name: nginx
-spec:
-  selector:
-    matchLabels:
-      app.kubernetes.io/name: nginx
-  template:
-    metadata:
-      labels:
-        app.kubernetes.io/name: nginx
-    spec:
-      containers:
-        - name: nginx
-          image: {{ DOCKER_IMAGE_NGINX }}
-          volumeMounts:
-            - mountPath: /etc/nginx/conf.d/
-              name: config
-            {{ patch("k8s-deployments-nginx-volume-mounts")|indent(12) }}
-          ports:
-            - containerPort: 80
-      volumes:
-        - name: config
-          configMap:
-            name: nginx-config
-        {{ patch("k8s-deployments-nginx-volumes")|indent(8) }}
 {% if RUN_REDIS %}
 ---
 apiVersion: apps/v1

tutor/templates/k8s/services.yml

@@ -1,4 +1,4 @@
-{% if RUN_CADDY %}
+{% if ENABLE_WEB_PROXY %}
 ---
 apiVersion: v1
 kind: Service
@@ -98,18 +98,6 @@ spec:
   selector:
     app.kubernetes.io/name: mysql
 {% endif %}
----
-apiVersion: v1
-kind: Service
-metadata:
-  name: nginx
-spec:
-  type: NodePort
-  ports:
-    - port: 80
-      name: http
-  selector:
-    app.kubernetes.io/name: nginx
 {% if RUN_REDIS %}
 ---
 apiVersion: v1

tutor/templates/k8s/volumes.yml

@@ -1,4 +1,4 @@
-{% if RUN_CADDY %}
+{% if ENABLE_WEB_PROXY %}
 ---
 apiVersion: v1
 kind: PersistentVolumeClaim

tutor/templates/kustomization.yml

@@ -34,9 +34,6 @@ configMapGenerator:
 - name: openedx-config
   files:{% for file in "apps/openedx/config"|walk_templates %}
   - {{ file }}{% endfor %}
-- name: nginx-config
-  files:{% for file in "apps/nginx"|walk_templates %}
-  - {{ file }}{% endfor %}
 - name: redis-config
   files:
   - apps/redis/redis.conf

tutor/templates/local/docker-compose.prod.yml

@@ -1,36 +1,23 @@
 version: "3.7"
 services:
-  {% if RUN_CADDY %}
-  # Web proxy for SSL termination
+  # Web proxy for load balancing and SSL termination
   caddy:
     image: {{ DOCKER_IMAGE_CADDY }}
     restart: unless-stopped
     ports:
-      - "80:80"
-      {% if ENABLE_HTTPS %}- "443:443"{% endif %}
+      - "{{ CADDY_HTTP_PORT }}:80"
+      {% if ENABLE_HTTPS and ENABLE_WEB_PROXY %}- "443:443"{% endif %}
     volumes:
       - ../apps/caddy/Caddyfile:/etc/caddy/Caddyfile:ro
-      {% if ENABLE_HTTPS %}- ../../data/caddy:/data{% endif %}
-  {% endif %}
-
-  # Web server
-  nginx:
-    image: {{ DOCKER_IMAGE_NGINX }}
-    restart: unless-stopped
-    {% if not RUN_CADDY %}
-    ports:
-      - "{{ NGINX_HTTP_PORT }}:80"
-    {% endif %}
-    {% if RUN_CADDY and not ENABLE_HTTPS %}
+      {% if ENABLE_HTTPS and ENABLE_WEB_PROXY %}- ../../data/caddy:/data{% endif %}
+    {% if not ENABLE_HTTPS %}
     networks:
       default:
-        # These aliases are for internal communication between containers when running locally with *.local.overhang.io hostnames.
+        # These aliases are for internal communication between containers when running locally
+        # with *.local.overhang.io hostnames.
         aliases:
           - "{{ LMS_HOST }}"
-          {{ patch("local-docker-compose-nginx-aliases")|indent(10) }}
+          {{ patch("local-docker-compose-caddy-aliases")|indent(10) }}
     {% endif %}
-    volumes:
-      - ../apps/nginx:/etc/nginx/conf.d/:ro
-    depends_on: {{ [("lms", RUN_LMS), ("cms", RUN_CMS)]|list_if }}
 
-  {{ patch("local-docker-compose-prod-services")|indent(2) }}
+  {{ patch("local-docker-compose-prod-services")|indent(2) }}