feat: upgrade to Maple

- A shared cookie domain between lms and cms is no longer recommended:
https://github.com/edx/edx-platform/blob/master/docs/guides/studio_oauth.rst
- refactor: clean mounted data folder in lms/cms. In Lilac, the
bind-mounted lms/data and cms/data folders are a mess because new
folders are created there for every new course organisation.  These
folders are empty. As far as we know they are useless... With this
change we move these folders to a dedicated "modulestore" subdirectory;
which corresponds better to the initial intent of the fs_root setting.
- fix: frontend failure during login to the lms. See:
https://github.com/openedx/build-test-release-wg/issues/104
- feat: move all forum-related code to a dedicated plugin. Forum is an
optional feature, and as such it deserves its own plugin. Starting from
Maple, users will be able to install the forum from
https://github.com/overhangio/tutor-forum/
- migrate from DCS_* session cookie settings to SESSION_*. That's
because edx-platform no longer depends on django-cookies-samesite. Close
https://github.com/openedx/build-test-release-wg/issues/110
- get rid of tons of deprecation warnings in the lms/cms
- feat: make it possible to point to themed assets. Cherry-picking this
change makes it possible to point to themed assets with a theme-agnostic
url, notably from MFEs.
- Install all official plugins as part of the `tutor[full]` package.
- Don't print error messages about loading plugins during autocompletion.
- Prompt for image building when upgrading from one release to the next.
- Add `tutor local start --skip-build` option to skip building Docker images.

Close #450.
Close #545.
This commit is contained in:
Régis Behmo 2021-10-18 11:43:40 +02:00
parent 62ddc01cdc
commit 7c157eccd5
34 changed files with 231 additions and 121 deletions

View File

@ -1,26 +1,3 @@
# Changelog (nightly branch)
Note: Breaking changes between versions are indicated by "💥".
- [Feature] Better support of Caddy as a load balancer in Kubernetes:
- Make it possible to start/stop a selection of resources with ``tutor k8s start/stop [names...]``.
- Make it easy to deploy an independent LoadBalancer by converting the caddy service to a ClusterIP when ``ENABLE_WEB_PROXY=false``.
- Add a ``app.kubernetes.io/component: loadbalancer`` label to the LoadBalancer service.
- Add ``app.kubernetes.io/name`` labels to all services.
- Preserve the LoadBalancer service in ``tutor k8s stop`` commands.
- Wait for the caddy deployment to be ready before running initialisation jobs.
- [Security] On Kubernetes, convert all NodePort services to ClusterIP to guarantee network isolation from outside the cluster.
- 💥[Improvement] Drop Python 3.5 compatibility.
- [Bugfix] Fix docker-compose project name in development on nightly branch.
- 💥[Bugfix] No longer track the Tutor version number in resource labels (and label selectors, which breaks the update of Deployment resources), but instead do so in resource annotations.
- [Bugfix] Make it possible for plugins to implement the "caddyfile" patch without relying on the "port" local variable.
- 💥[Improvement] Move the Open edX forum to a [dedicated plugin](https://github.com/overhangio/tutor-forum/) (#450).
- 💥[Improvement] Get rid of the "tutor-openedx" package, which is no longer supported.
- [Bugfix] Fix running Caddy container in k8s, which should always be the case even if `ENABLE_WEB_PROXY` is false.
- 💥[Improvement] Run all services as unprivileged containers, for better security. This has multiple consequences:
- The "openedx-dev" image is now built with `tutor dev dc build lms`.
- The "smtp" service now runs the "devture/exim-relay" Docker image, which is unprivileged. Also, the default SMTP port is now 8025.
- 💥[Feature] Get rid of the nginx container and service, which is now replaced by Caddy. this has the following consequences:
- Patches "nginx-cms", "nginx-lms", "nginx-extra", "local-docker-compose-nginx-aliases" are replaced by "caddyfile-cms", "caddyfile-lms", "caddyfile", " local-docker-compose-caddy-aliases".
- Patches "k8s-deployments-nginx-volume-mounts", "k8s-deployments-nginx-volumes" were obsolete and are removed.
- The `NGINX_HTTP_PORT` setting is renamed to `CADDY_HTTP_PORT`.

View File

@ -4,6 +4,32 @@ Note: Breaking changes between versions are indicated by "💥".
## Unreleased
- 💥[Improvement] Upgrade to Maple
- Install all official plugins as part of the `tutor[full]` package.
- Don't print error messages about loading plugins during autocompletion.
- Prompt for image building when upgrading from one release to the next.
- 💥 Allow concurrent logins to the LMS and the CMS.
- Add `tutor local start --skip-build` option to skip building Docker images.
- [Feature] Better support of Caddy as a load balancer in Kubernetes:
- Make it possible to start/stop a selection of resources with ``tutor k8s start/stop [names...]``.
- Make it easy to deploy an independent LoadBalancer by converting the caddy service to a ClusterIP when ``ENABLE_WEB_PROXY=false``.
- Add a ``app.kubernetes.io/component: loadbalancer`` label to the LoadBalancer service.
- Add ``app.kubernetes.io/name`` labels to all services.
- Preserve the LoadBalancer service in ``tutor k8s stop`` commands.
- Wait for the caddy deployment to be ready before running initialisation jobs.
- Fix running Caddy container in k8s, which should always be the case even if `ENABLE_WEB_PROXY` is false.
- [Security] On Kubernetes, convert all NodePort services to ClusterIP to guarantee network isolation from outside the cluster.
- 💥[Improvement] Move the Open edX forum to a [dedicated plugin](https://github.com/overhangio/tutor-forum/) (#450).
- 💥[Improvement] Drop Python 3.5 compatibility.
- 💥[Bugfix] No longer track the Tutor version number in resource labels (and label selectors, which breaks the update of Deployment resources), but instead do so in resource annotations.
- 💥[Improvement] Get rid of the "tutor-openedx" package, which is no longer supported.
- 💥[Improvement] Run all services as unprivileged containers, for better security. This has multiple consequences:
- The "openedx-dev" image is now built with `tutor dev dc build lms`.
- The "smtp" service now runs the "devture/exim-relay" Docker image, which is unprivileged. Also, the default SMTP port is now 8025.
- 💥[Feature] Get rid of the nginx container and service, which is now replaced by Caddy. this has the following consequences:
- Patches "nginx-cms", "nginx-lms", "nginx-extra", "local-docker-compose-nginx-aliases" are replaced by "caddyfile-cms", "caddyfile-lms", "caddyfile", " local-docker-compose-caddy-aliases".
- Patches "k8s-deployments-nginx-volume-mounts", "k8s-deployments-nginx-volumes" were obsolete and are removed.
- The `NGINX_HTTP_PORT` setting is renamed to `CADDY_HTTP_PORT`.
- [Bugfix] Fix building of the `openedx` image on ARM64 due to missing `libgeos-dev`
## v12.2.0 (2021-12-08)

View File

@ -79,7 +79,7 @@ You may want to pull/push images from/to a custom docker registry. For instance,
Open edX customisation
~~~~~~~~~~~~~~~~~~~~~~
- ``OPENEDX_COMMON_VERSION`` (default: ``"open-release/lilac.2"``)
- ``OPENEDX_COMMON_VERSION`` (default: ``"open-release/maple.beta2"``)
This defines the default version that will be pulled from all Open edX git repositories.
@ -281,16 +281,16 @@ Note that your edx-platform version must be a fork of the latest release **tag**
If you don't create your fork from this tag, you *will* have important compatibility issues with other services. In particular:
- Do not try to run a fork from an older (pre-Lilac) version of edx-platform: this will simply not work.
- Do not try to run a fork from an older (pre-Maple) version of edx-platform: this will simply not work.
- Do not try to run a fork from the edx-platform master branch: there is a 99% probability that it will fail.
- Do not try to run a fork from the open-release/lilac.master branch: Tutor will attempt to apply security and bug fix patches that might already be included in the open-release/lilac.master but which were not yet applied to the latest release tag. Patch application will thus fail if you base your fork from the open-release/lilac.master branch.
- Do not try to run a fork from the open-release/maple.master branch: Tutor will attempt to apply security and bug fix patches that might already be included in the open-release/maple.master but which were not yet applied to the latest release tag. Patch application will thus fail if you base your fork from the open-release/maple.master branch.
.. _i18n:
Adding custom translations
~~~~~~~~~~~~~~~~~~~~~~~~~~
If you are not running Open edX in English, chances are that some strings will not be properly translated. In most cases, this is because not enough contributors have helped translate Open edX in your language. It happens! With Tutor, available translated languages include those that come bundled with `edx-platform <https://github.com/edx/edx-platform/tree/open-release/lilac.master/conf/locale>`__ as well as those from `openedx-i18n <https://github.com/openedx/openedx-i18n/tree/master/edx-platform/locale>`__.
If you are not running Open edX in English, chances are that some strings will not be properly translated. In most cases, this is because not enough contributors have helped translate Open edX in your language. It happens! With Tutor, available translated languages include those that come bundled with `edx-platform <https://github.com/edx/edx-platform/tree/open-release/maple.master/conf/locale>`__ as well as those from `openedx-i18n <https://github.com/openedx/openedx-i18n/tree/master/edx-platform/locale>`__.
Tutor offers a relatively simple mechanism to add custom translations to the openedx Docker image. You should create a folder that corresponds to your language code in the "build/openedx/locale" folder of the Tutor environment. This folder should contain a "LC_MESSAGES" folder. For instance::
@ -311,9 +311,9 @@ Then, add a "django.po" file there that will contain your custom translations::
.. warning::
Don't forget to specify the file ``Content-Type`` when adding message strings with non-ASCII characters; otherwise a ``UnicodeDecodeError`` will be raised during compilation.
The "String to translate" part should match *exactly* the string that you would like to translate. You cannot make it up! The best way to find this string is to copy-paste it from the `upstream django.po file for the English language <https://github.com/edx/edx-platform/blob/open-release/lilac.master/conf/locale/en/LC_MESSAGES/django.po>`__.
The "String to translate" part should match *exactly* the string that you would like to translate. You cannot make it up! The best way to find this string is to copy-paste it from the `upstream django.po file for the English language <https://github.com/edx/edx-platform/blob/open-release/maple.master/conf/locale/en/LC_MESSAGES/django.po>`__.
If you cannot find the string to translate in this file, then it means that you are trying to translate a string that is used in some piece of javascript code. Those strings are stored in a different file named "djangojs.po". You can check it out `in the edx-platform repo as well <https://github.com/edx/edx-platform/blob/open-release/lilac.master/conf/locale/en/LC_MESSAGES/djangojs.po>`__. Your custom javascript strings should also be stored in a "djangojs.po" file that should be placed in the same directory.
If you cannot find the string to translate in this file, then it means that you are trying to translate a string that is used in some piece of javascript code. Those strings are stored in a different file named "djangojs.po". You can check it out `in the edx-platform repo as well <https://github.com/edx/edx-platform/blob/open-release/maple.master/conf/locale/en/LC_MESSAGES/djangojs.po>`__. Your custom javascript strings should also be stored in a "djangojs.po" file that should be placed in the same directory.
To recap, here is an example. To translate a few strings in French, both from django.po and djangojs.po, we would have the following file hierarchy::

View File

@ -25,7 +25,7 @@ This ``openedx-dev`` development image differs from the ``openedx`` production i
- The user that runs inside the container has the same UID as the user on the host, in order to avoid permission problems inside mounted volumes (and in particular in the edx-platform repository).
- Additional python and system requirements are installed for convenient debugging: `ipython <https://ipython.org/>`__, `ipdb <https://pypi.org/project/ipdb/>`__, vim, telnet.
- The edx-platform `development requirements <https://github.com/edx/edx-platform/blob/open-release/lilac.master/requirements/edx/development.in>`__ are installed.
- The edx-platform `development requirements <https://github.com/edx/edx-platform/blob/open-release/maple.master/requirements/edx/development.in>`__ are installed.
Since the ``openedx-dev`` is based upon the ``openedx`` docker image, it should be re-built every time the ``openedx`` docker image is modified.
@ -137,7 +137,7 @@ Following the instructions :ref:`above <bind_mounts>` on how to bind-mount direc
If you choose any but the first solution above, you will have to make sure that your fork works with Tutor.
First of all, you should make sure that you are working off the ``open-release/lilac.2`` tag. See the :ref:`fork edx-platform section <edx_platform_fork>` for more information.
First of all, you should make sure that you are working off the ``open-release/maple.beta2`` tag. See the :ref:`fork edx-platform section <edx_platform_fork>` for more information.
Then, you should run the following commands::

View File

@ -1,3 +1,3 @@
.. parsed-literal::
pip install tutor
pip install tutor[full]

View File

@ -38,7 +38,7 @@ The `devstack <https://github.com/edx/devstack>`_ is meant for development only,
Is Tutor officially supported by edX?
-------------------------------------
As of the Open edX Lilac release (June 9th 2021), Tutor is one of the two officially supported installation methods for Open edX: see the `official installation instructions <https://edx.readthedocs.io/projects/edx-installing-configuring-and-running/en/open-release-lilac.master/installation/index.html>`__. We expect that by Maple (December 9th 2021) the native installation will be deprecated and Tutor will become the only officially recommended installation method, unless major issues are discovered. However, Tutor remains developed independently from edX, both by its parent company Overhang.IO and the :ref:`project maintainers <maintainers>`.
Yes: as of the Open edX Maple release (December 9th 2021), Tutor is the only officially supported installation methods for Open edX: see the `official installation instructions <https://edx.readthedocs.io/projects/edx-installing-configuring-and-running/en/open-release-maple.master/installation/index.html>`__.
What features are missing from Tutor?
-------------------------------------

View File

@ -6,7 +6,7 @@ Getting started
.. toctree::
:maxdepth: 2
install
intro
quickstart
install
whatnext
whatnext

View File

@ -1,14 +1,15 @@
.. _install:
Install Tutor
=============
Installing Tutor
================
.. _requirements:
Requirements
------------
* Supported OS: Tutor runs on any 64-bit, UNIX-based system. It was also reported to work on Windows.
* Supported OS: Tutor runs on any 64-bit, UNIX-based OS. It was also reported to work on Windows (with `WSL 2 <https://docs.microsoft.com/en-us/windows/wsl/install>`__).
* Architecture: support for ARM64 is a work-in-progress. See `this issue <https://github.com/overhangio/tutor/issues/510>`__.
* Required software:
- `Docker <https://docs.docker.com/engine/installation/>`__: v18.06.0+
@ -26,46 +27,48 @@ Requirements
.. note::
On Mac OS, by default, containers are allocated 2 GB of RAM, which is not enough. You should follow `these instructions from the official Docker documentation <https://docs.docker.com/docker-for-mac/#advanced>`__ to allocate at least 4-5 GB to the Docker daemon. If the deployment fails because of insufficient memory during database migrations, check the :ref:`relevant section in the troubleshooting guide <migrations_killed>`.
Download
--------
Choose **one** of the installation methods below. If you install Tutor in different ways, you will end up with multiple ``tutor`` executables, which is going to be very confusing. At any time, you can check the path to your ``tutor`` executable by running ``which tutor``.
Python package
~~~~~~~~~~~~~~
.. include:: download/pip.rst
Check the "tutor" package on Pypi: https://pypi.org/project/tutor. You will python >= 3.6 with pip and the libyaml development headers. On Ubuntu, these requirements can be installed by running::
sudo apt install python3 python3-pip libyaml-dev
.. _install_binary:
Direct binary download
----------------------
Binary release
~~~~~~~~~~~~~~
The latest binaries can be downloaded from https://github.com/overhangio/tutor/releases. From the command line:
.. include:: download/binary.rst
This is the simplest and recommended installation method for most people. Note however that you will not be able to use custom plugins with this pre-compiled binary. The only plugins you can use with this approach are those that are already bundled with the binary: see the :ref:`existing plugins <existing_plugins>`.
This is the simplest and recommended installation method for most people who do not have Python 3 on their machine. Note however that **you will not be able to use custom plugins** with this pre-compiled binary. The only plugins you can use with this approach are those that are already bundled with the binary: see the :ref:`existing plugins <existing_plugins>`.
.. _install_source:
Alternative installation methods
--------------------------------
If you would like to inspect the Tutor source code, you are most welcome to install Tutor from `Pypi <https://pypi.org/project/tutor/>`_ or directly from `the Github repository <https://github.com/overhangio/tutor>`_. You will need python >= 3.6 with pip and the libyaml development headers. On Ubuntu, these requirements can be installed by running::
sudo apt install python3 python3-pip libyaml-dev
Installing from pypi
~~~~~~~~~~~~~~~~~~~~
.. include:: download/pip.rst
Installing from source
~~~~~~~~~~~~~~~~~~~~~~
::
To inspect the Tutor source code, install Tutor from `the Github repository <https://github.com/overhangio/tutor>`__::
git clone https://github.com/overhangio/tutor
cd tutor
pip install -e .
DNS records
-----------
Configuring DNS records
-----------------------
When running a server in production, it is necessary to define `DNS records <https://en.wikipedia.org/wiki/Domain_Name_System#Resource_records>`__ which will make it possible to access your Open edX platform by name in your browser. The precise procedure to create DNS records vary from one provider to the next and is beyond the scope of these docs. You should create a record of type A with a name equal to your LMS hostname (given by ``tutor config printvalue LMS_HOST``) and a value that indicates the IP address of your server. Applications other than the LMS, such as the studio, ecommerce, etc. typically reside in subdomains of the LMS. Thus, you should also create a CNAME record to point all subdomains of the LMS to the LMS_HOST.
For instance, the demo Open edX server that runs at http://demo.openedx.overhang.io has the following DNS records::
For instance, the demo Open edX server that runs at https://demo.openedx.overhang.io has the following DNS records::
demo.openedx 1800 IN A 172.105.89.208
*.demo.openedx 1800 IN CNAME demo.openedx.overhang.io.
@ -73,7 +76,7 @@ For instance, the demo Open edX server that runs at http://demo.openedx.overhang
.. _cloud_install:
Zero-click AWS installation
~~~~~~~~~~~~~~~~~~~~~~~~~~~
---------------------------
Tutor can be launched on Amazon Web Services very quickly with the `official Tutor AMI <https://aws.amazon.com/marketplace/pp/B07PV3TB8X>`__. Shell access is not required, as all configuration will happen through the Tutor web user interface. For detailed installation instructions, we recommend watching the following video:
@ -86,14 +89,14 @@ Upgrading
With Tutor, it is very easy to upgrade to a more recent Open edX or Tutor release. Just install the latest ``tutor`` version (using either methods above) and run the ``quickstart`` command again. If you have :ref:`customised <configuration_customisation>` your docker images, you will have to re-build them prior to running ``quickstart``.
``quickstart`` should take care of automatically running the upgrade process. If for some reason you need to *manually* upgrade from an Open edX release to the next, you should run ``tutor local upgrade``. For instance, to upgrade from Koa to Lilac, run::
``quickstart`` should take care of automatically running the upgrade process. If for some reason you need to *manually* upgrade from an Open edX release to the next, you should run ``tutor local upgrade``. For instance, to upgrade from Lilac to Maple, run::
tutor local upgrade --from=koa
tutor local upgrade --from=lilac
.. _autocomplete:
Autocomplete
------------
Shell autocompletion
--------------------
Tutor is built on top of `Click <https://click.palletsprojects.com>`_, which is a great library for building command line interface (CLI) tools. As such, Tutor benefits from all Click features, including `auto-completion <https://click.palletsprojects.com/en/8.x/bashcomplete/>`_. After installing Tutor, auto-completion can be enabled in bash by running::

View File

@ -61,7 +61,7 @@ A demo Open edX platform is available at https://demo.openedx.overhang.io. This
* Admin user: username=admin email=admin@overhang.io password=admin
* Student user: username=student email=student@overhang.io password=student
The Android mobile application for this website can be downloaded at this url: http://demo.openedx.overhang.io/static/mobile/app.apk
The Android mobile application for this demo platform can be downloaded at this url: https://mobile.demo.openedx.overhang.io/app.apk
Urls:

View File

@ -22,8 +22,8 @@ Yes :) This is what happens when you run ``tutor local quickstart``:
2. Configuration files are generated from templates.
3. Docker images are downloaded.
4. Docker containers are provisioned.
5. A full, production-ready Open edX platform (`Lilac <https://edx.readthedocs.io/projects/edx-installing-configuring-and-running/en/open-release-lilac.master/platform_releases/koa.html>`__ release) is run with docker-compose.
5. A full, production-ready Open edX platform (`Maple <https://edx.readthedocs.io/projects/edx-installing-configuring-and-running/en/open-release-maple.master/platform_releases/maple.html>`__ release) is run with docker-compose.
The whole procedure should require less than 10 minutes, on a server with a good bandwidth. Note that your host environment will not be affected in any way, since everything runs inside docker containers. Root access is not even necessary.
There's a lot more to Tutor than that! To learn more about what you can do with Tutor and Open edX, check out the :ref:`whatnext` section. If the quickstart installation method above somehow didn't work for you, check out the :ref:`troubleshooting` guide.
There's a lot more to Tutor than that! To learn more about what you can do with Tutor and Open edX, check out the :ref:`whatnext` section. If the quickstart installation method above somehow didn't work for you, check out the :ref:`troubleshooting` guide.

View File

@ -9,7 +9,7 @@ Portainer is a web UI for managing docker containers. It lets you view your enti
--volume=/var/run/docker.sock:/var/run/docker.sock \
--volume=/tmp/portainer:/data \
-p 9000:9000 \
portainer/portainer:latest --bind=:9000
portainer/portainer-ce:latest --bind=:9000
You can then view the portainer UI at `http://localhost:9000 <http://localhost:9000>`_. You will be asked to define a password for the admin user. Then, select a "Local environment" to work on; hit "Connect" and select the "local" group to view all running containers.

View File

@ -1,12 +1,12 @@
# change version ranges when upgrading from lilac
tutor-android>=12.0.0,<13.0.0
tutor-discovery>=12.0.0,<13.0.0
tutor-ecommerce>=12.0.0,<13.0.0
tutor-forum>=12.0.0,<13.0.0
tutor-license>=12.0.0,<13.0.0
tutor-mfe>=12.0.0,<13.0.0
tutor-minio>=12.0.0,<13.0.0
tutor-notes>=12.0.0,<13.0.0
tutor-richie>=12.0.0,<13.0.0
tutor-webui>=12.0.0,<13.0.0
tutor-xqueue>=12.0.0,<13.0.0
# change version ranges when upgrading from maple
tutor-android>=13.0.0,<14.0.0
tutor-discovery>=13.0.0,<14.0.0
tutor-ecommerce>=13.0.0,<14.0.0
tutor-forum>=13.0.0,<14.0.0
tutor-license>=13.0.0,<14.0.0
tutor-mfe>=13.0.0,<14.0.0
tutor-minio>=13.0.0,<14.0.0
tutor-notes>=13.0.0,<14.0.0
tutor-richie>=13.0.0,<14.0.0
tutor-webui>=13.0.0,<14.0.0
tutor-xqueue>=13.0.0,<14.0.0

View File

@ -9,7 +9,9 @@ def load_readme():
with io.open(os.path.join(HERE, "README.rst"), "rt", encoding="utf8") as f:
readme = f.read()
# Replace img src for publication on pypi
return readme.replace("./docs/img/", "https://github.com/overhangio/tutor/raw/master/docs/img/")
return readme.replace(
"./docs/img/", "https://github.com/overhangio/tutor/raw/master/docs/img/"
)
def load_about():
@ -21,9 +23,9 @@ def load_about():
return about
def load_requirements():
def load_requirements(filename: str):
with io.open(
os.path.join(HERE, "requirements", "base.in"), "rt", encoding="utf-8"
os.path.join(HERE, "requirements", filename), "rt", encoding="utf-8"
) as f:
return [line.strip() for line in f if is_requirement(line)]
@ -53,7 +55,10 @@ setup(
packages=find_packages(exclude=["tests*"]),
include_package_data=True,
python_requires=">=3.6",
install_requires=load_requirements(),
install_requires=load_requirements("base.in"),
extras_require={
"full": load_requirements("plugins.txt"),
},
entry_points={"console_scripts": ["tutor=tutor.commands.cli:main"]},
classifiers=[
"Development Status :: 5 - Production/Stable",

View File

@ -2,7 +2,7 @@ import os
# Increment this version number to trigger a new release. See
# docs/tutor.html#versioning for information on the versioning scheme.
__version__ = "12.2.0"
__version__ = "13.0.0"
# The version suffix will be appended to the actual version, separated by a
# dash. Use this suffix to differentiate between the actual released version and

View File

@ -66,11 +66,16 @@ class BaseComposeContext(BaseJobContext):
short_help="Run all or a selection of services.",
help="Run all or a selection of services. Docker images will be rebuilt where necessary.",
)
@click.option("--skip-build", is_flag=True, help="Skip image building")
@click.option("-d", "--detach", is_flag=True, help="Start in daemon mode")
@click.argument("services", metavar="service", nargs=-1)
@click.pass_obj
def start(context: BaseComposeContext, detach: bool, services: List[str]) -> None:
command = ["up", "--remove-orphans", "--build"]
def start(
context: BaseComposeContext, skip_build: bool, detach: bool, services: List[str]
) -> None:
command = ["up", "--remove-orphans"]
if not skip_build:
command.append("--build")
if detach:
command.append("-d")

View File

@ -320,7 +320,8 @@ def delete(context: Context, yes: bool) -> None:
def init(context: Context, limit: Optional[str]) -> None:
config = tutor_config.load(context.root)
runner = K8sJobRunner(context.root, config)
for name in ["caddy", "elasticsearch", "mysql", "mongodb"]:
wait_for_pod_ready(config, "caddy")
for name in ["elasticsearch", "mysql", "mongodb"]:
if tutor_config.is_service_activated(config, name):
wait_for_pod_ready(config, name)
jobs.initialise(runner, limit_to=limit)
@ -442,7 +443,7 @@ def wait(context: Context, name: str) -> None:
"--from",
"from_version",
default="koa",
type=click.Choice(["ironwood", "juniper", "koa"]),
type=click.Choice(["ironwood", "juniper", "koa", "lilac"]),
)
@click.pass_obj
def upgrade(context: Context, from_version: str) -> None:
@ -461,6 +462,10 @@ def upgrade(context: Context, from_version: str) -> None:
upgrade_from_koa(config)
running_version = "lilac"
if running_version == "lilac":
# Nothing to do here
running_version = "maple"
def upgrade_from_ironwood(config: Config) -> None:
if not config["RUN_MONGODB"]:

View File

@ -104,7 +104,7 @@ Your Open edX platform is ready and can be accessed at the following urls:
"--from",
"from_version",
default="koa",
type=click.Choice(["ironwood", "juniper", "koa"]),
type=click.Choice(["ironwood", "juniper", "koa", "lilac"]),
)
@click.option("-I", "--non-interactive", is_flag=True, help="Run non-interactively")
@click.pass_context
@ -137,6 +137,21 @@ Are you sure you want to continue?"""
upgrade_from_koa(context, config)
running_version = "lilac"
if running_version == "lilac":
# Nothing to do here
running_version = "maple"
if not non_interactive:
question = f"""
Your platform was successfuly upgraded from {from_version} to {running_version}. Depending on your setup, you might have to rebuild some of your Docker images. You can do this now by running the following command in a different shell:
tutor images build openedx # add your custom images here
Press enter when you are ready to continue"""
click.confirm(
fmt.question(question), default=True, abort=True, prompt_suffix=" "
)
def upgrade_from_ironwood(context: click.Context, config: Config) -> None:
click.echo(fmt.title("Upgrading from Ironwood"))

View File

@ -38,7 +38,6 @@ def load_full(root: str) -> Config:
"""
Load a full configuration, with user, base and defaults.
"""
convert_json2yml(root)
config = get_user(root)
update_with_base(config)
update_with_defaults(config)
@ -84,6 +83,7 @@ def get_user(root: str) -> Config:
Overrides from environment variables are loaded as well.
"""
convert_json2yml(root)
path = config_path(root)
config = {}
if os.path.exists(path):

View File

@ -331,9 +331,14 @@ def current_release(root: str) -> str:
"""
Return the name of the current Open edX release.
"""
return {"0": "ironwood", "3": "ironwood", "10": "juniper", "11": "koa"}[
current_version(root).split(".")[0]
]
return {
"0": "ironwood",
"3": "ironwood",
"10": "juniper",
"11": "koa",
"12": "lilac",
"13": "maple",
}[current_version(root).split(".")[0]]
def current_version(root: str) -> str:

View File

@ -1,3 +1,5 @@
import os
import click
STDOUT = None
@ -45,4 +47,8 @@ def alert(text: str) -> str:
def echo(text: str, err: bool = False) -> None:
if os.environ.get("_TUTOR_COMPLETE"):
if os.environ.get("COMP_WORDS") or os.environ.get("COMP_CWORD"):
# Don't even attempt to log stuff when we are actually auto-completing shell commands.
return
click.echo(text, file=STDOUT, err=err)

View File

@ -37,7 +37,7 @@
"EMAIL_USE_TLS": {{ "true" if SMTP_USE_TLS else "false" }},
"HTTPS": "{{ "on" if ENABLE_HTTPS else "off" }}",
"LANGUAGE_CODE": "{{ LANGUAGE_CODE }}",
"SESSION_COOKIE_DOMAIN": ".{{ LMS_HOST|common_domain(CMS_HOST) }}",
"SESSION_COOKIE_DOMAIN": "{{ CMS_HOST }}",
{{ patch("cms-env", separator=",\n", suffix=",")|indent(2) }}
"CACHES": {
"default": {

View File

@ -45,7 +45,7 @@
"ACE_ROUTING_KEY": "edx.lms.core.default",
"HTTPS": "{{ "on" if ENABLE_HTTPS else "off" }}",
"LANGUAGE_CODE": "{{ LANGUAGE_CODE }}",
"SESSION_COOKIE_DOMAIN": ".{{ LMS_HOST|common_domain(CMS_HOST) }}",
"SESSION_COOKIE_DOMAIN": "{{ LMS_HOST }}",
{{ patch("lms-env", separator=",\n", suffix=",")|indent(2) }}
"CACHES": {
"default": {

View File

@ -4,6 +4,11 @@ from cms.envs.devstack import *
LMS_BASE = "{{ LMS_HOST }}:8000"
LMS_ROOT_URL = "http://" + LMS_BASE
# Authentication
SOCIAL_AUTH_EDX_OAUTH2_KEY = "{{ CMS_OAUTH2_KEY_SSO_DEV }}"
SOCIAL_AUTH_EDX_OAUTH2_PUBLIC_URL_ROOT = LMS_ROOT_URL
FEATURES["PREVIEW_LMS_BASE"] = "{{ PREVIEW_LMS_HOST }}:8000"
{% include "apps/openedx/settings/partials/common_cms.py" %}

View File

@ -9,4 +9,8 @@ ALLOWED_HOSTS = [
"cms",
]
# Authentication
SOCIAL_AUTH_EDX_OAUTH2_KEY = "{{ CMS_OAUTH2_KEY_SSO }}"
SOCIAL_AUTH_EDX_OAUTH2_PUBLIC_URL_ROOT = "{% if ENABLE_HTTPS %}https{% else %}http{% endif %}://{{ LMS_HOST }}"
{{ patch("openedx-cms-production-settings") }}

View File

@ -17,6 +17,9 @@ CMS_BASE = "{{ CMS_HOST}}:8001"
CMS_ROOT_URL = "http://{}".format(CMS_BASE)
LOGIN_REDIRECT_WHITELIST.append(CMS_BASE)
# CMS authentication
IDA_LOGOUT_URI_LIST.append("http://{{ CMS_HOST }}:8001/complete/logout")
FEATURES['ENABLE_COURSEWARE_MICROFRONTEND'] = False
LOGGING["loggers"]["oauth2_provider"] = {

View File

@ -15,14 +15,17 @@ ALLOWED_HOSTS = [
# Chrome to support samesite=none cookies.
SESSION_COOKIE_SECURE = True
CSRF_COOKIE_SECURE = True
DCS_SESSION_COOKIE_SAMESITE = "None"
SESSION_COOKIE_SAMESITE = "None"
{% else %}
# When we cannot provide secure session/csrf cookies, we must disable samesite=none
SESSION_COOKIE_SECURE = False
CSRF_COOKIE_SECURE = False
DCS_SESSION_COOKIE_SAMESITE = "Lax"
SESSION_COOKIE_SAMESITE = "Lax"
{% endif %}
# CMS authentication
IDA_LOGOUT_URI_LIST.append("{% if ENABLE_HTTPS %}https{% else %}http{% endif %}://{{ CMS_HOST }}/complete/logout")
# Required to display all courses on start page
SEARCH_SKIP_ENROLLMENT_START_DATE_FILTERING = True

View File

@ -25,7 +25,8 @@ CONTENTSTORE = {
}
# Load module store settings from config files
update_module_store_settings(MODULESTORE, doc_store_settings=DOC_STORE_CONFIG)
DATA_DIR = "/openedx/data/"
DATA_DIR = "/openedx/data/modulestore"
for store in MODULESTORE["default"]["OPTIONS"]["stores"]:
store["OPTIONS"]["fs_root"] = DATA_DIR
@ -96,8 +97,11 @@ LOGGING["handlers"]["tracking"] = {
}
LOGGING["loggers"]["tracking"]["handlers"] = ["console", "local", "tracking"]
# Silence some loggers (note: we must attempt to get rid of these when upgrading from one release to the next)
import warnings
warnings.filterwarnings("ignore", category=DeprecationWarning, module="newrelic.console")
from django.utils.deprecation import RemovedInDjango40Warning, RemovedInDjango41Warning
warnings.filterwarnings("ignore", category=RemovedInDjango40Warning)
warnings.filterwarnings("ignore", category=RemovedInDjango41Warning)
warnings.filterwarnings("ignore", category=DeprecationWarning, module="lms.djangoapps.course_wiki.plugins.markdownedx.wiki_plugin")
warnings.filterwarnings("ignore", category=DeprecationWarning, module="wiki.plugins.links.wiki_plugin")
@ -154,6 +158,7 @@ JWT_AUTH["JWT_ISSUERS"] = [
# Enable/Disable some features globally
FEATURES["ENABLE_DISCUSSION_SERVICE"] = False
FEATURES["PREVENT_CONCURRENT_LOGINS"] = False
# Disable codejail support
# explicitely configuring python is necessary to prevent unsafe calls
@ -165,11 +170,5 @@ CODE_JAIL = {
"user": None,
}
# Custom features
# LTI 1.3 will be enabled by default after lilac, and it's going to be a big
# deal, so we enable it early. We should remove this once the feature flag is
# deprecated.
FEATURES["LTI_1P3_ENABLED"] = True
{{ patch("openedx-common-settings") }}
######## End of settings common to LMS and CMS

View File

@ -3,6 +3,12 @@
######## Common CMS settings
STUDIO_NAME = u"{{ PLATFORM_NAME }} - Studio"
# Authentication
SOCIAL_AUTH_EDX_OAUTH2_SECRET = "{{ CMS_OAUTH2_SECRET }}"
SOCIAL_AUTH_EDX_OAUTH2_URL_ROOT = "http://lms:8000"
SOCIAL_AUTH_REDIRECT_IS_HTTPS = False # scheme is correctly included in redirect_uri
MAX_ASSET_UPLOAD_FILE_SIZE_IN_MB = 100
FRONTEND_LOGIN_URL = LMS_ROOT_URL + '/login'
@ -16,4 +22,4 @@ for folder in [LOG_DIR, MEDIA_ROOT, STATIC_ROOT_BASE]:
{{ patch("openedx-cms-common-settings") }}
######## End of common CMS settings
######## End of common CMS settings

View File

@ -14,9 +14,9 @@ RUN apt update && \
apt install -y libssl-dev zlib1g-dev libbz2-dev \
libreadline-dev libsqlite3-dev wget curl llvm libncurses5-dev libncursesw5-dev \
xz-utils tk-dev libffi-dev liblzma-dev python-openssl git
ARG PYTHON_VERSION=3.8.6
ARG PYTHON_VERSION=3.8.12
ENV PYENV_ROOT /opt/pyenv
RUN git clone https://github.com/pyenv/pyenv $PYENV_ROOT --branch v1.2.21 --depth 1
RUN git clone https://github.com/pyenv/pyenv $PYENV_ROOT --branch v2.2.2 --depth 1
RUN $PYENV_ROOT/bin/pyenv install $PYTHON_VERSION
RUN $PYENV_ROOT/versions/$PYTHON_VERSION/bin/python -m venv /openedx/venv
@ -44,6 +44,12 @@ RUN git config --global user.email "tutor@overhang.io" \
{{ patch("openedx-dockerfile-git-patches-default") }}
{% else %}
# Patch edx-platform
# Fix language cookie "samesite" attribute
# https://github.com/edx/edx-platform/pull/29621
RUN git fetch --depth=2 https://github.com/regisb/edx-platform 51e0ec3b97ae5badbf947d53ac07bd5496c10cde && git cherry-pick 51e0ec3b97ae5badbf947d53ac07bd5496c10cde
# Fix forum notification for questions
# https://github.com/edx/edx-platform/pull/29611
RUN git fetch --depth=2 https://github.com/open-craft/edx-platform/ 03731f19459e558f188c06aac5cc9ca1bbc675c2 && git cherry-pick 03731f19459e558f188c06aac5cc9ca1bbc675c2
{% endif %}
{# Example: RUN git fetch --depth=2 https://github.com/edx/edx-platform <GITSHA1> && git cherry-pick <GITSHA1> #}
@ -82,7 +88,7 @@ RUN pip install -r ./requirements/edx/base.txt
RUN pip install django-redis==4.12.1
# Install uwsgi
RUN pip install uwsgi==2.0.19.1
RUN pip install uwsgi==2.0.20
{{ patch("openedx-dockerfile-post-python-requirements") }}

View File

@ -1 +1 @@
EDX_PLATFORM_REVISION: lilac
EDX_PLATFORM_REVISION: maple

View File

@ -1,6 +1,10 @@
---
CMS_OAUTH2_SECRET: "{{ 24|random_string }}"
ID: "{{ 24|random_string }}"
JWT_RSA_PRIVATE_KEY: "{{ 2048|rsa_private_key }}"
MYSQL_ROOT_PASSWORD: "{{ 8|random_string }}"
OPENEDX_MYSQL_PASSWORD: "{{ 8|random_string }}"
OPENEDX_SECRET_KEY: "{{ 24|random_string }}"
PLUGINS:
# The MFE plugin is required
- mfe

View File

@ -4,20 +4,21 @@
# This must be defined early
CADDY_HTTP_PORT: 80
CMS_HOST: "studio.{{ LMS_HOST }}"
CMS_OAUTH2_KEY_SSO: "cms-sso"
CMS_OAUTH2_KEY_SSO_DEV: "cms-sso-dev"
CONTACT_EMAIL: "contact@{{ LMS_HOST }}"
DEV_PROJECT_NAME: "{{ TUTOR_APP }}_dev"
DOCKER_REGISTRY: "docker.io/"
DOCKER_IMAGE_OPENEDX: "{{ DOCKER_REGISTRY }}overhangio/openedx:{{ TUTOR_VERSION }}"
DOCKER_IMAGE_OPENEDX_DEV: "openedx-dev"
DOCKER_IMAGE_CADDY: "{{ DOCKER_REGISTRY }}caddy:2.3.0"
DOCKER_IMAGE_ELASTICSEARCH: "{{ DOCKER_REGISTRY }}elasticsearch:7.10.1"
DOCKER_IMAGE_MONGODB: "{{ DOCKER_REGISTRY }}mongo:4.2.17"
DOCKER_IMAGE_MYSQL: "{{ DOCKER_REGISTRY }}mysql:5.7.35"
DOCKER_IMAGE_ELASTICSEARCH: "{{ DOCKER_REGISTRY }}elasticsearch:7.10.1"
DOCKER_IMAGE_NGINX: "{{ DOCKER_REGISTRY }}nginx:1.21.1"
DOCKER_IMAGE_CADDY: "docker.io/caddy:2.3.0"
DOCKER_IMAGE_ELASTICSEARCH: "docker.io/elasticsearch:7.10.1"
DOCKER_IMAGE_MONGODB: "docker.io/mongo:4.2.17"
DOCKER_IMAGE_MYSQL: "docker.io/mysql:5.7.35"
DOCKER_IMAGE_PERMISSIONS: "{{ DOCKER_REGISTRY }}overhangio/openedx-permissions:{{ TUTOR_VERSION }}"
DOCKER_IMAGE_REDIS: "{{ DOCKER_REGISTRY }}redis:6.2.6"
DOCKER_IMAGE_SMTP: "{{ DOCKER_REGISTRY }}devture/exim-relay:4.94.2-r0-4"
DOCKER_IMAGE_REDIS: "docker.io/redis:6.2.6"
DOCKER_IMAGE_SMTP: "docker.io/devture/exim-relay:4.94.2-r0-4"
LOCAL_PROJECT_NAME: "{{ TUTOR_APP }}_local"
ELASTICSEARCH_HOST: "elasticsearch"
ELASTICSEARCH_PORT: 9200
ELASTICSEARCH_SCHEME: "http"
@ -45,14 +46,13 @@ OPENEDX_LMS_UWSGI_WORKERS: 2
OPENEDX_MYSQL_DATABASE: "openedx"
OPENEDX_CSMH_MYSQL_DATABASE: "{{ OPENEDX_MYSQL_DATABASE }}_csmh"
OPENEDX_MYSQL_USERNAME: "openedx"
OPENEDX_COMMON_VERSION: "open-release/lilac.3"
OPENEDX_COMMON_VERSION: "open-release/maple.1"
OPENEDX_EXTRA_PIP_REQUIREMENTS:
- "openedx-scorm-xblock<13.0.0,>=12.0.0"
- "openedx-scorm-xblock<14.0.0,>=13.0.0"
MYSQL_HOST: "mysql"
MYSQL_PORT: 3306
MYSQL_ROOT_USERNAME: "root"
PLATFORM_NAME: "My Open edX"
PLUGINS: []
PREVIEW_LMS_HOST: "preview.{{ LMS_HOST }}"
REDIS_HOST: "redis"
REDIS_PORT: 6379

View File

@ -4,6 +4,27 @@ echo "Loading settings $DJANGO_SETTINGS_MODULE"
./manage.py lms migrate
# Create oauth2 apps for CMS SSO
# https://github.com/edx/edx-platform/blob/master/docs/guides/studio_oauth.rst
./manage.py lms manage_user cms cms@openedx --unusable-password
./manage.py lms create_dot_application \
--grant-type authorization-code \
--redirect-uris "{% if ENABLE_HTTPS %}https{% else %}http{% endif %}://{{ CMS_HOST }}/complete/edx-oauth2/" \
--client-id {{ CMS_OAUTH2_KEY_SSO }} \
--client-secret {{ CMS_OAUTH2_SECRET }} \
--scopes user_id \
--skip-authorization \
--update cms-sso cms
./manage.py lms create_dot_application \
--grant-type authorization-code \
--redirect-uris "http://{{ CMS_HOST }}:8001/complete/edx-oauth2/" \
--client-id {{ CMS_OAUTH2_KEY_SSO_DEV }} \
--client-secret {{ CMS_OAUTH2_SECRET }} \
--scopes user_id \
--skip-authorization \
--update cms-sso-dev cms
# Fix incorrect uploaded file path
if [ -d /openedx/data/uploads/ ]; then
if [ -n "$(ls -A /openedx/data/uploads/)" ]; then

View File

@ -33,15 +33,27 @@ configMapGenerator:
- name: openedx-settings-lms
files:{% for file in "apps/openedx/settings/lms"|walk_templates %}
- {{ file }}{% endfor %}
options:
labels:
app.kubernetes.io/name: openedx
- name: openedx-settings-cms
files:{% for file in "apps/openedx/settings/cms"|walk_templates %}
- {{ file }}{% endfor %}
options:
labels:
app.kubernetes.io/name: openedx
- name: openedx-config
files:{% for file in "apps/openedx/config"|walk_templates %}
- {{ file }}{% endfor %}
options:
labels:
app.kubernetes.io/name: openedx
- name: redis-config
files:
- apps/redis/redis.conf
options:
labels:
app.kubernetes.io/name: redis
{{ patch("kustomization-configmapgenerator") }}
{{ patch("kustomization") }}
{{ patch("kustomization") }}