Merge branch 'nightly' into olive
This commit is contained in:
Régis Behmo
commit
5ef246479b
2
Makefile
2
Makefile
|
|
@ -40,7 +40,7 @@ test-unit: ## Run unit tests
|
|||
python -m unittest discover tests
|
||||
|
||||
test-types: ## Check type definitions
|
||||
mypy --exclude=templates --ignore-missing-imports --strict ${SRC_DIRS}
|
||||
mypy --exclude=templates --ignore-missing-imports --implicit-reexport --strict ${SRC_DIRS}
|
||||
|
||||
test-pythonpackage: build-pythonpackage ## Test that package can be uploaded to pypi
|
||||
twine check dist/tutor-$(shell make version).tar.gz
|
||||
|
|
|
|||
|
|
@ -55,16 +55,17 @@ Features
|
|||
* No technical skill required with the `zero-click Tutor AWS image <https://docs.tutor.overhang.io/install.html#zero-click-aws-installation>`__
|
||||
|
||||
.. _readme_intro_end:
|
||||
|
||||
..
|
||||
TODO: replace image + alt with tutor local launch
|
||||
.. image:: ./docs/img/quickstart.gif
|
||||
:alt: Tutor local quickstart
|
||||
:target: https://terminalizer.com/view/91b0bfdd557
|
||||
|
||||
Quickstart
|
||||
Launch
|
||||
----------
|
||||
|
||||
1. Install the `latest stable release <https://github.com/overhangio/tutor/releases>`_ of Tutor
|
||||
2. Run ``tutor local quickstart``
|
||||
2. Run ``tutor local launch``
|
||||
3. You're done!
|
||||
|
||||
Documentation
|
||||
|
|
|
|||
|
|
@ -0,0 +1,15 @@
|
|||
- [Improvement] Add the `-h` help option to all commands and subcommands. Previously, we could only use `--help`, which is quite long for lazy fingers. (by @regisb)
|
||||
- 💥[Feature] Add an extensible `local/dev/k8s do ...` command to trigger custom job commands. These commands are used to run a series of bash scripts in designated containers. Any plugin can add custom jobs thanks to the `CLI_DO_COMMANDS` filter. This causes the following breaking changes:
|
||||
- The "init", "createuser", "settheme", "importdemocourse" commands were all migrated to this new interface. For instance, `tutor local init` was replaced by `tutor local do init`.
|
||||
- Plugin developers are encouraged to replace calls to the `COMMANDS_INIT` and `COMMANDS_PRE_INIT` filters by `CLI_DO_INIT_TASKS`.
|
||||
- [Feature] Implement hook filter priorities, which work like action priorities. (by @regisb)
|
||||
- 💥[Improvement] Remove the `local/dev bindmount` commands, which have been marked as deprecated for some time. The `--mount` option should be used instead.
|
||||
- 💥[Bugfix] Fix local installation requirements. Plugins that implemented the "openedx-dockerfile-post-python-requirements" patch and that needed access to the edx-platform repo will no longer work. Instead, these plugins should implement the "openedx-dockerfile-pre-assets" patch. This scenario should be very rare, though. (by @regisb)
|
||||
- 💥[Improvement] Rename the implementation of tutor <mode> quickstart to tutor <mode> launch. (by @Carlos-Muniz)
|
||||
- 💥[Improvement] Remove the implementation of tutor dev runserver. (by @Carlos-Muniz)
|
||||
- [Bugfix] Fix MongoDB replica set connection error resulting from edx-platform's pymongo (3.10.1 -> 3.12.3) upgrade ([edx-platform#30569](https://github.com/openedx/edx-platform/pull/30569)). (by @ormsbee)
|
||||
- [Improvement] For Tutor Nightly (and only Nightly), official plugins are now installed from their nightly branches on GitHub instead of a version range on PyPI. This will allow Nightly users to install all official plugins by running ``pip install -e ".[full]"``.
|
||||
- [Bugfix] Remove edX references from bulk emails ([issue](https://github.com/openedx/build-test-release-wg/issues/100)).
|
||||
- [Bugfix] Update ``celery`` invocations for lms-worker and cms-worker to be compatible with Celery 5 CLI.
|
||||
- [Improvement] Point CMS at its config file using ``CMS_CFG`` environment variable instead of deprecated ``STUDIO_CFG``.
|
||||
- [Bugfix] Start MongoDB when running migrations, because a new data migration fails if MongoDB is not running
|
||||
|
|
@ -15,7 +15,7 @@ This section does not cover :ref:`plugin development <plugins>`. For simple chan
|
|||
Configuration
|
||||
-------------
|
||||
|
||||
With Tutor, all Open edX deployment parameters are stored in a single ``config.yml`` file. This is the file that is generated when you run ``tutor local quickstart`` or ``tutor config save``. To view the content of this file, run::
|
||||
With Tutor, all Open edX deployment parameters are stored in a single ``config.yml`` file. This is the file that is generated when you run ``tutor local launch`` or ``tutor config save``. To view the content of this file, run::
|
||||
|
||||
cat "$(tutor config printroot)/config.yml"
|
||||
|
||||
|
|
@ -330,7 +330,7 @@ The following sections describe how to modify various aspects of the docker imag
|
|||
|
||||
tutor local stop
|
||||
|
||||
The custom image will be used the next time you run ``tutor local quickstart`` or ``tutor local start``. Do not attempt to run ``tutor local restart``! Restarting will not pick up the new image and will continue to use the old image.
|
||||
The custom image will be used the next time you run ``tutor local launch`` or ``tutor local start``. Do not attempt to run ``tutor local restart``! Restarting will not pick up the new image and will continue to use the old image.
|
||||
|
||||
openedx Docker Image build arguments
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
|
@ -486,7 +486,7 @@ By default, Tutor runs the `overhangio/openedx <https://hub.docker.com/r/overhan
|
|||
|
||||
(See the relevant :ref:`configuration parameters <docker_images>`.)
|
||||
|
||||
The customised Docker image tag value will then be used by Tutor to run the platform, for instance when running ``tutor local quickstart``.
|
||||
The customised Docker image tag value will then be used by Tutor to run the platform, for instance when running ``tutor local launch``.
|
||||
|
||||
|
||||
Passing custom docker build options
|
||||
|
|
|
|||
29
docs/dev.rst
29
docs/dev.rst
|
|
@ -13,7 +13,7 @@ First, ensure you have already :ref:`installed Tutor <install>` (for development
|
|||
|
||||
Then, launch the developer platform setup process::
|
||||
|
||||
tutor dev quickstart
|
||||
tutor dev launch
|
||||
|
||||
This will perform several tasks for you. It will:
|
||||
|
||||
|
|
@ -51,14 +51,14 @@ To bring down your platform's containers, simply run::
|
|||
Starting the platform back up
|
||||
-----------------------------
|
||||
|
||||
Once you have used ``quickstart`` once, you can start the platform in the future with the lighter-weight ``start`` command, which brings up containers but does not perform any initialization tasks::
|
||||
Once you have used ``launch`` once, you can start the platform in the future with the lighter-weight ``start`` command, which brings up containers but does not perform any initialization tasks::
|
||||
|
||||
tutor dev start # Run platform in the same terminal ("attached")
|
||||
tutor dev start -d # Or, run platform the in the background ("detached")
|
||||
|
||||
Nonetheless, ``quickstart`` is idempotent, so it is always safe to run it again in the future without risk to your data. In fact, you may find it useful to use this command as a one-stop-shop for pulling images, running migrations, initializing new plugins you have enabled, and/or executing any new initialization steps that may have been introduced since you set up Tutor::
|
||||
Nonetheless, ``launch`` is idempotent, so it is always safe to run it again in the future without risk to your data. In fact, you may find it useful to use this command as a one-stop-shop for pulling images, running migrations, initializing new plugins you have enabled, and/or executing any new initialization steps that may have been introduced since you set up Tutor::
|
||||
|
||||
tutor dev quickstart --pullimages
|
||||
tutor dev launch --pullimages
|
||||
|
||||
|
||||
Running arbitrary commands
|
||||
|
|
@ -112,7 +112,7 @@ It may sometimes be convenient to mount container directories on the host, for i
|
|||
Bind-mount volumes with ``--mount``
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
The ``quickstart``, ``run``, ``init`` and ``start`` subcommands of ``tutor dev`` and ``tutor local`` support the ``-m/--mount`` option (see :option:`tutor dev start -m`) which can take two different forms. The first is explicit::
|
||||
The ``launch``, ``run``, ``init`` and ``start`` subcommands of ``tutor dev`` and ``tutor local`` support the ``-m/--mount`` option (see :option:`tutor dev start -m`) which can take two different forms. The first is explicit::
|
||||
|
||||
tutor dev start --mount=lms:/path/to/edx-platform:/openedx/edx-platform lms
|
||||
|
||||
|
|
@ -152,24 +152,6 @@ Then, bind-mount that folder back in the container with the ``--mount`` option (
|
|||
|
||||
You can then edit the files in ``~/venv`` on your local filesystem and see the changes live in your container.
|
||||
|
||||
Bind-mount from the "volumes/" directory
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. warning:: Bind-mounting volumes with the ``bindmount`` command is no longer the default, recommended way of bind-mounting volumes from the host. Instead, see the :ref:`mount option <mount_option>` and the ``tutor dev/local copyfrom`` commands.
|
||||
|
||||
Tutor makes it easy to create a bind-mount from an existing container. First, copy the contents of a container directory with the ``bindmount`` command. For instance, to copy the virtual environment of the "lms" container::
|
||||
|
||||
tutor dev bindmount lms /openedx/venv
|
||||
|
||||
This command recursively copies the contents of the ``/opendedx/venv`` directory to ``$(tutor config printroot)/volumes/venv``. The code of any Python dependency can then be edited -- for instance, you can then add a ``breakpoint()`` statement for step-by-step debugging, or implement a custom feature.
|
||||
|
||||
Then, bind-mount the directory back in the container with the ``--mount`` option::
|
||||
|
||||
tutor dev start --mount=lms:$(tutor config printroot)/volumes/venv:/openedx/venv lms
|
||||
|
||||
.. note::
|
||||
The ``bindmount`` command and the ``--mount=...`` option syntax are available both for the ``tutor local`` and ``tutor dev`` commands.
|
||||
|
||||
Manual bind-mount to any directory
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
|
|
@ -283,6 +265,7 @@ Then, run unit tests with ``pytest`` commands::
|
|||
export EDXAPP_TEST_MONGO_HOST=mongodb
|
||||
pytest common
|
||||
pytest openedx
|
||||
pytest xmodule
|
||||
|
||||
# Run tests on LMS
|
||||
export DJANGO_SETTINGS_MODULE=lms.envs.tutor.test
|
||||
|
|
|
|||
|
|
@ -8,5 +8,5 @@ Getting started
|
|||
|
||||
install
|
||||
intro
|
||||
quickstart
|
||||
quickstart
|
||||
whatnext
|
||||
|
|
|
|||
|
|
@ -1,7 +1,8 @@
|
|||
.. include:: ../README.rst
|
||||
:start-after: _readme_intro_start:
|
||||
:end-before: _readme_intro_end:
|
||||
|
||||
..
|
||||
TODO replace quickstart.gif + alt with 'launch'
|
||||
.. image:: ./img/quickstart.gif
|
||||
:alt: Tutor local quickstart
|
||||
:target: https://terminalizer.com/view/91b0bfdd557
|
||||
|
|
|
|||
|
|
@ -91,20 +91,20 @@ To upgrade Open edX or benefit from the latest features and bug fixes, you shoul
|
|||
|
||||
pip install --upgrade "tutor[full]"
|
||||
|
||||
Then run the ``quickstart`` command again. Depending on your deployment target, run one of::
|
||||
Then run the ``launch`` command again. Depending on your deployment target, run one of::
|
||||
|
||||
tutor local quickstart # for local installations
|
||||
tutor dev quickstart # for local development installations
|
||||
tutor k8s quickstart # for Kubernetes installation
|
||||
tutor local launch # for local installations
|
||||
tutor dev launch # for local development installations
|
||||
tutor k8s launch # for Kubernetes installation
|
||||
|
||||
Upgrading with custom Docker images
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
If you run :ref:`customised <configuration_customisation>` Docker images, you need to rebuild them before running ``quickstart``::
|
||||
If you run :ref:`customised <configuration_customisation>` Docker images, you need to rebuild them before running ``launch``::
|
||||
|
||||
tutor config save
|
||||
tutor images build all # specify here the images that you need to build
|
||||
tutor local quickstart
|
||||
tutor local launch
|
||||
|
||||
Upgrading to a new Open edX release
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
|
@ -121,12 +121,12 @@ Major Open edX releases are published twice a year, in June and December, by the
|
|||
4. Test the new release in a sandboxed environment.
|
||||
5. If you are running edx-platform, or some other repository from a custom branch, then you should rebase (and test) your changes on top of the latest release tag (see :ref:`edx_platform_fork`).
|
||||
|
||||
The process for upgrading from one major release to the next works similarly to any other upgrade, with the ``quickstart`` command (see above). The single difference is that if the ``quickstart`` command detects that your tutor environment was generated with an older release, it will perform a few release-specific upgrade steps. These extra upgrade steps will be performed just once. But they will be ignored if you updated your local environment (for instance: with ``tutor config save``) before running ``quickstart``. This situation typically occurs if you need to re-build some Docker images (see above). In such a case, you should make use of the ``upgrade`` command. For instance, to upgrade a local installation from Maple to Nutmeg and rebuild some Docker images, run::
|
||||
The process for upgrading from one major release to the next works similarly to any other upgrade, with the ``launch`` command (see above). The single difference is that if the ``launch`` command detects that your tutor environment was generated with an older release, it will perform a few release-specific upgrade steps. These extra upgrade steps will be performed just once. But they will be ignored if you updated your local environment (for instance: with ``tutor config save``) before running ``launch``. This situation typically occurs if you need to re-build some Docker images (see above). In such a case, you should make use of the ``upgrade`` command. For instance, to upgrade a local installation from Maple to Nutmeg and rebuild some Docker images, run::
|
||||
|
||||
tutor config save
|
||||
tutor images build all # list the images that should be rebuilt here
|
||||
tutor local upgrade --from=maple
|
||||
tutor local quickstart
|
||||
tutor local launch
|
||||
|
||||
.. _autocomplete:
|
||||
|
||||
|
|
|
|||
|
|
@ -99,7 +99,7 @@ Because the Tutor environment is generated entirely from the values in ``config.
|
|||
|
||||
You can now take advantage of the Tutor-powered CLI (item #3) to bootstrap your Open edX platform::
|
||||
|
||||
tutor local quickstart
|
||||
tutor local launch
|
||||
|
||||
Under the hood, Tutor simply runs ``docker-compose`` and ``docker`` commands to launch your platform. These commands are printed in the standard output, such that you are free to replicate the same behaviour by simply copying/pasting the same commands.
|
||||
|
||||
|
|
@ -117,7 +117,7 @@ as well as command trees for each mode in which Tutor can run::
|
|||
tutor k8s ... # Commands for managing a Kubernetes Open edX deployment.
|
||||
tutor dev ... # Commands for hacking on Open edX in development mode.
|
||||
|
||||
Within each mode, Tutor has subcommands for managing that type of Open edX instance. Many of them are common between modes, such as ``quickstart``, ``start``, ``stop``, ``exec``, and ``logs``. For example::
|
||||
Within each mode, Tutor has subcommands for managing that type of Open edX instance. Many of them are common between modes, such as ``launch``, ``start``, ``stop``, ``exec``, and ``logs``. For example::
|
||||
|
||||
tutor local logs # View logs of a local deployment.
|
||||
tutor k8s logs # View logs of a Kubernetes-managed deployment.
|
||||
|
|
|
|||
|
|
@ -85,19 +85,19 @@ The other benefit of ``kubectl apply`` is that it allows you to customise the Ku
|
|||
|
||||
To learn more about "kustomizations", refer to the `official documentation <https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/>`__.
|
||||
|
||||
Quickstart
|
||||
Launch
|
||||
----------
|
||||
|
||||
Launch the platform on Kubernetes in one command::
|
||||
|
||||
tutor k8s quickstart
|
||||
tutor k8s launch
|
||||
|
||||
All Kubernetes resources are associated with the "openedx" namespace. If you don't see anything in the Kubernetes dashboard, you are probably looking at the wrong namespace... 😉
|
||||
|
||||
.. image:: img/k8s-dashboard.png
|
||||
:alt: Kubernetes dashboard ("openedx" namespace)
|
||||
|
||||
The same ``tutor k8s quickstart`` command can be used to upgrade the cluster to the latest version.
|
||||
The same ``tutor k8s launch`` command can be used to upgrade the cluster to the latest version.
|
||||
|
||||
Other commands
|
||||
--------------
|
||||
|
|
|
|||
|
|
@ -18,7 +18,7 @@ In the following, environment and data files will be generated in a user-specifi
|
|||
tutor run ...
|
||||
|
||||
.. note::
|
||||
As of v10.0.0, a locally-running Open edX platform can no longer be accessed from http://localhost or http://studio.localhost. Instead, when running ``tutor local quickstart``, you must now decide whether you are running a platform that will be used in production. If not, the platform will be automatically be bound to http://local.overhang.io and http://studio.local.overhang.io, which are domain names that point to 127.0.0.1 (localhost). This change was made to facilitate internal communication between Docker containers.
|
||||
As of v10.0.0, a locally-running Open edX platform can no longer be accessed from http://localhost or http://studio.localhost. Instead, when running ``tutor local launch``, you must now decide whether you are running a platform that will be used in production. If not, the platform will be automatically be bound to http://local.overhang.io and http://studio.local.overhang.io, which are domain names that point to 127.0.0.1 (localhost). This change was made to facilitate internal communication between Docker containers.
|
||||
|
||||
Main commands
|
||||
-------------
|
||||
|
|
@ -32,9 +32,9 @@ All-in-one command
|
|||
|
||||
A fully-functional platform can be configured and run in one command::
|
||||
|
||||
tutor local quickstart
|
||||
tutor local launch
|
||||
|
||||
But you may want to run commands one at a time: it's faster when you need to run only part of the local deployment process, and it helps you understand how your platform works. In the following, we decompose the ``quickstart`` command.
|
||||
But you may want to run commands one at a time: it's faster when you need to run only part of the local deployment process, and it helps you understand how your platform works. In the following, we decompose the ``launch`` command.
|
||||
|
||||
Configuration
|
||||
~~~~~~~~~~~~~
|
||||
|
|
@ -80,7 +80,7 @@ Service initialisation
|
|||
|
||||
::
|
||||
|
||||
tutor local init
|
||||
tutor local do init
|
||||
|
||||
This command should be run just once. It will initialise all applications in a running platform. In particular, this will create the required databases tables and apply database migrations for all applications.
|
||||
|
||||
|
|
@ -120,7 +120,7 @@ Creating a new user with staff and admin rights
|
|||
|
||||
You will most certainly need to create a user to administer the platform. Just run::
|
||||
|
||||
tutor local createuser --staff --superuser yourusername user@email.com
|
||||
tutor local do createuser --staff --superuser yourusername user@email.com
|
||||
|
||||
You will be asked to set the user password interactively.
|
||||
|
||||
|
|
@ -131,7 +131,7 @@ Importing the demo course
|
|||
|
||||
After a fresh installation, your platform will not have a single course. To import the `Open edX demo course <https://github.com/openedx/edx-demo-course>`_, run::
|
||||
|
||||
tutor local importdemocourse
|
||||
tutor local do importdemocourse
|
||||
|
||||
.. _settheme:
|
||||
|
||||
|
|
@ -140,7 +140,7 @@ Setting a new theme
|
|||
|
||||
The default Open edX theme is rather bland, so Tutor makes it easy to switch to a different theme::
|
||||
|
||||
tutor local settheme mytheme
|
||||
tutor local do settheme mytheme
|
||||
|
||||
Out of the box, only the default "open-edx" theme is available. We also developed `Indigo, a beautiful, customizable theme <https://github.com/overhangio/indigo>`__ which is easy to install with Tutor.
|
||||
|
||||
|
|
|
|||
|
|
@ -11,7 +11,7 @@ Tutor comes with a plugin system that allows anyone to customise the deployment
|
|||
# 2) Enable the plugin
|
||||
tutor plugins enable myapp
|
||||
# 3) Reconfigure and restart the platform
|
||||
tutor local quickstart
|
||||
tutor local launch
|
||||
|
||||
For simple changes, it may be extremely easy to create a Tutor plugin: even non-technical users may get started with our :ref:`plugin_development_tutorial` tutorial. We also provide a list of :ref:`simple example plugins <plugins_examples>`.
|
||||
|
||||
|
|
|
|||
|
|
@ -51,7 +51,7 @@ You should be able to view your changes in every LMS and CMS settings file::
|
|||
|
||||
Now just restart your platform to start sending tracking events to Google Analytics::
|
||||
|
||||
tutor local quickstart
|
||||
tutor local launch
|
||||
|
||||
That's it! And it's very easy to share your plugins. Just upload them to your Github repo and share the url with other users. They will be able to install your plugin by running::
|
||||
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
.. _quickstart:
|
||||
.. _launch:
|
||||
|
||||
Quickstart (1-click install)
|
||||
----------------------------
|
||||
|
|
@ -11,12 +11,12 @@ Or `download <https://github.com/overhangio/tutor/releases>`_ the pre-compiled b
|
|||
|
||||
.. include:: download/binary.rst
|
||||
|
||||
2. Run ``tutor local quickstart``
|
||||
2. Run ``tutor local launch``
|
||||
3. You're done!
|
||||
|
||||
**That's it?**
|
||||
|
||||
Yes :) This is what happens when you run ``tutor local quickstart``:
|
||||
Yes :) This is what happens when you run ``tutor local launch``:
|
||||
|
||||
1. You answer a few questions about the :ref:`configuration` of your Open edX platform.
|
||||
2. Configuration files are generated from templates.
|
||||
|
|
@ -26,4 +26,4 @@ Yes :) This is what happens when you run ``tutor local quickstart``:
|
|||
|
||||
The whole procedure should require less than 10 minutes, on a server with good bandwidth. Note that your host environment will not be affected in any way, since everything runs inside docker containers. Root access is not even necessary.
|
||||
|
||||
There's a lot more to Tutor than that! To learn more about what you can do with Tutor and Open edX, check out the :ref:`whatnext` section. If the quickstart installation method above somehow didn't work for you, check out the :ref:`troubleshooting` guide.
|
||||
There's a lot more to Tutor than that! To learn more about what you can do with Tutor and Open edX, check out the :ref:`whatnext` section. If the launch installation method above somehow didn't work for you, check out the :ref:`troubleshooting` guide.
|
||||
|
|
|
|||
|
|
@ -63,15 +63,15 @@ If the above command does not work, you should fix your Docker installation. Som
|
|||
"Running migrations... Killed!" / "Command failed with status 137: docker-compose"
|
||||
----------------------------------------------------------------------------------
|
||||
|
||||
Open edX requires at least 4 GB RAM, in particular, to run the SQL migrations. If the ``tutor local quickstart`` command dies after displaying "Running migrations", you most probably need to buy more memory or add swap to your machine.
|
||||
Open edX requires at least 4 GB RAM, in particular, to run the SQL migrations. If the ``tutor local launch`` command dies after displaying "Running migrations", you most probably need to buy more memory or add swap to your machine.
|
||||
|
||||
On macOS, by default, Docker allocates at most 2 GB of RAM to containers. ``quickstart`` tries to check your current allocation and outputs a warning if it can't find a value of at least 4 GB. You should follow `these instructions from the official Docker documentation <https://docs.docker.com/docker-for-mac/#advanced>`__ to allocate at least 4-5 GB to the Docker daemon.
|
||||
On macOS, by default, Docker allocates at most 2 GB of RAM to containers. ``launch`` tries to check your current allocation and outputs a warning if it can't find a value of at least 4 GB. You should follow `these instructions from the official Docker documentation <https://docs.docker.com/docker-for-mac/#advanced>`__ to allocate at least 4-5 GB to the Docker daemon.
|
||||
|
||||
If migrations were killed halfway, there is a good chance that the MySQL database is in a state that is hard to recover from. The easiest way to recover is simply to delete all the MySQL data and restart the quickstart process. After you have allocated more memory to the Docker daemon, run::
|
||||
If migrations were killed halfway, there is a good chance that the MySQL database is in a state that is hard to recover from. The easiest way to recover is simply to delete all the MySQL data and restart the launch process. After you have allocated more memory to the Docker daemon, run::
|
||||
|
||||
tutor local stop
|
||||
sudo rm -rf "$(tutor config printroot)/data/mysql"
|
||||
tutor local quickstart
|
||||
tutor local launch
|
||||
|
||||
.. warning::
|
||||
THIS WILL ERASE ALL YOUR DATA! Do not run this on a production instance. This solution is only viable for new Open edX installations.
|
||||
|
|
|
|||
|
|
@ -48,9 +48,9 @@ Finish setup and start Tutor
|
|||
From this point on, use Tutor as normal. For example, start Open edX and run migrations with::
|
||||
|
||||
tutor local start -d
|
||||
tutor local init
|
||||
tutor local do init
|
||||
|
||||
Or for a development environment::
|
||||
|
||||
tutor dev start -d
|
||||
tutor dev init
|
||||
tutor dev do init
|
||||
|
|
|
|||
|
|
@ -34,7 +34,7 @@ Don't forget to replace your email address and password in the prompt above. If
|
|||
|
||||
Then, restart your platform::
|
||||
|
||||
$ tutor local quickstart
|
||||
$ tutor local launch
|
||||
|
||||
That's it! You can send a test email with the following command::
|
||||
|
||||
|
|
|
|||
|
|
@ -22,7 +22,7 @@ In addition to installing Tutor Nightly itself, this will install automatically
|
|||
|
||||
Once Tutor Nightly is installed, you can run the usual ``tutor`` commands::
|
||||
|
||||
tutor dev quickstart
|
||||
tutor dev launch
|
||||
tutor dev run lms bash
|
||||
# ... and so on
|
||||
|
||||
|
|
|
|||
|
|
@ -4,9 +4,9 @@ Upgrading from older releases
|
|||
Upgrading from v3+
|
||||
~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Just upgrade Tutor using your :ref:`favorite installation method <install>` and run quickstart again::
|
||||
Just upgrade Tutor using your :ref:`favorite installation method <install>` and run launch again::
|
||||
|
||||
tutor local quickstart
|
||||
tutor local launch
|
||||
|
||||
Upgrading from v1 or v2
|
||||
~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
|
@ -22,4 +22,4 @@ Then, install Tutor using one of the :ref:`installation methods <install>`. Then
|
|||
|
||||
Finally, launch your platform with::
|
||||
|
||||
tutor local quickstart
|
||||
tutor local launch
|
||||
|
|
|
|||
|
|
@ -218,7 +218,7 @@ You can now run the "myservice" container which will execute the ``CMD`` stateme
|
|||
Declaring initialisation tasks
|
||||
------------------------------
|
||||
|
||||
Services often need to run specific tasks before they can be started. For instance, the LMS and the CMS need to apply database migrations. These commands are written in shell scripts that are executed whenever we run ``quickstart``. We call these scripts "init tasks". To add a new local init task, we must first add the corresponding service to the ``docker-compose-jobs.yml`` file by implementing the :patch:`local-docker-compose-jobs-services` patch::
|
||||
Services often need to run specific tasks before they can be started. For instance, the LMS and the CMS need to apply database migrations. These commands are written in shell scripts that are executed whenever we run ``launch``. We call these scripts "init tasks". To add a new local initialisation task, we must first add the corresponding service to the ``docker-compose-jobs.yml`` file by implementing the :patch:`local-docker-compose-jobs-services` patch::
|
||||
|
||||
hooks.Filters.ENV_PATCHES.add_item(
|
||||
(
|
||||
|
|
@ -234,24 +234,22 @@ The patch above defined the "myservice-job" container which will run our initial
|
|||
|
||||
$ tutor config save
|
||||
|
||||
Next, we create the folder which will contain our init task script::
|
||||
Next, we create an initialisation task by adding an item to the :py:data:`tutor.hooks.Filters.CLI_DO_INIT_TASKS` filter::
|
||||
|
||||
$ mkdir "$(tutor plugins printroot)/templates/myplugin/tasks"
|
||||
|
||||
Edit ``$(tutor plugins printroot)/templates/myplugin/tasks/init.sh``::
|
||||
|
||||
hooks.Filters.CLI_DO_INIT_TASKS.add_item(
|
||||
(
|
||||
"myservice",
|
||||
"""
|
||||
echo "++++++ initialising my plugin..."
|
||||
echo "++++++ done!"
|
||||
|
||||
Add our init task script to the :py:data:`tutor.hooks.Filters.COMMANDS_INIT` filter::
|
||||
|
||||
hooks.Filters.COMMANDS_INIT.add_item(
|
||||
("myservice", ("myplugin", "tasks", "init.sh")),
|
||||
"""
|
||||
)
|
||||
)
|
||||
|
||||
Run this initialisation task with::
|
||||
|
||||
$ tutor local init --limit=myplugin
|
||||
$ tutor local do init --limit=myplugin
|
||||
...
|
||||
Running init task: myplugin/tasks/init.sh
|
||||
...
|
||||
|
|
@ -354,8 +352,14 @@ Eventually, our plugin is composed of the following files, all stored within the
|
|||
)
|
||||
hooks.Filters.IMAGES_PUSH.add_item(("myservice", "myservice:latest"))
|
||||
hooks.Filters.IMAGES_PULL.add_item(("myservice", "myservice:latest"))
|
||||
hooks.Filters.COMMANDS_INIT.add_item(
|
||||
("myservice", ("myplugin", "tasks", "init.sh")),
|
||||
hooks.Filters.CLI_DO_INIT_TASKS.add_item(
|
||||
(
|
||||
"myservice",
|
||||
"""
|
||||
echo "++++++ initialising my plugin..."
|
||||
echo "++++++ done!"
|
||||
"""
|
||||
)
|
||||
)
|
||||
|
||||
``templates/myplugin/build/myservice/Dockerfile``
|
||||
|
|
|
|||
|
|
@ -48,7 +48,7 @@ Then, run a local webserver::
|
|||
|
||||
The LMS can then be accessed at http://local.overhang.io:8000. You will then have to :ref:`enable that theme <settheme>`::
|
||||
|
||||
tutor dev settheme mythemename
|
||||
tutor dev do settheme mythemename
|
||||
|
||||
Watch the themes folders for changes (in a different terminal)::
|
||||
|
||||
|
|
|
|||
|
|
@ -3,7 +3,7 @@
|
|||
What next?
|
||||
==========
|
||||
|
||||
You have gone through the :ref:`Quickstart installation <quickstart>`: at this point, you should have a running Open edX platform. If you don't, please follow the instructions from the :ref:`Troubleshooting <troubleshooting>` section.
|
||||
You have gone through the :ref:`Launch installation <launch>`: at this point, you should have a running Open edX platform. If you don't, please follow the instructions from the :ref:`Troubleshooting <troubleshooting>` section.
|
||||
|
||||
Logging-in as administrator
|
||||
---------------------------
|
||||
|
|
|
|||
|
|
@ -1,11 +1,13 @@
|
|||
# change version ranges when upgrading from nutmeg
|
||||
tutor-android>=14.0.0,<15.0.0
|
||||
tutor-discovery>=14.0.0,<15.0.0
|
||||
tutor-ecommerce>=14.0.0,<15.0.0
|
||||
tutor-forum>=14.0.0,<15.0.0
|
||||
tutor-license>=14.0.0,<15.0.0
|
||||
tutor-mfe>=14.0.0,<15.0.0
|
||||
tutor-minio>=14.0.0,<15.0.0
|
||||
tutor-notes>=14.0.0,<15.0.0
|
||||
tutor-webui>=14.0.0,<15.0.0
|
||||
tutor-xqueue>=14.0.0,<15.0.0
|
||||
# For Tutor Nightly, we install plugins from their nightly branches instead of from PyPI
|
||||
# (except tutor-license, for which we just want the latest version from PyPI).
|
||||
tutor-android @ git+https://github.com/overhangio/tutor-android@nightly
|
||||
tutor-discovery @ git+https://github.com/overhangio/tutor-discovery@nightly
|
||||
tutor-ecommerce @ git+https://github.com/overhangio/tutor-ecommerce@nightly
|
||||
tutor-forum @ git+https://github.com/overhangio/tutor-forum@nightly
|
||||
tutor-license
|
||||
tutor-mfe @ git+https://github.com/overhangio/tutor-mfe@nightly
|
||||
tutor-minio @ git+https://github.com/overhangio/tutor-minio@nightly
|
||||
tutor-notes @ git+https://github.com/overhangio/tutor-notes@nightly
|
||||
tutor-webui @ git+https://github.com/overhangio/tutor-webui@nightly
|
||||
tutor-xqueue @ git+https://github.com/overhangio/tutor-xqueue@nightly
|
||||
|
||||
|
|
|
|||
|
|
@ -17,7 +17,9 @@ class TestCommandMixin:
|
|||
return TestCommandMixin.invoke_in_root(root, args)
|
||||
|
||||
@staticmethod
|
||||
def invoke_in_root(root: str, args: t.List[str]) -> click.testing.Result:
|
||||
def invoke_in_root(
|
||||
root: str, args: t.List[str], catch_exceptions: bool = True
|
||||
) -> click.testing.Result:
|
||||
"""
|
||||
Use this method for commands that all need to run in the same root:
|
||||
|
||||
|
|
@ -33,4 +35,6 @@ class TestCommandMixin:
|
|||
},
|
||||
mix_stderr=False,
|
||||
)
|
||||
return runner.invoke(cli, args, obj=TestContext(root))
|
||||
return runner.invoke(
|
||||
cli, args, obj=TestContext(root), catch_exceptions=catch_exceptions
|
||||
)
|
||||
|
|
|
|||
|
|
@ -1,7 +1,7 @@
|
|||
import os
|
||||
import unittest
|
||||
|
||||
from tests.helpers import TestContext, TestJobRunner, temporary_root
|
||||
from tests.helpers import TestContext, TestTaskRunner, temporary_root
|
||||
from tutor import config as tutor_config
|
||||
|
||||
|
||||
|
|
@ -15,4 +15,4 @@ class TestContextTests(unittest.TestCase):
|
|||
self.assertFalse(
|
||||
os.path.exists(os.path.join(context.root, tutor_config.CONFIG_FILENAME))
|
||||
)
|
||||
self.assertTrue(isinstance(runner, TestJobRunner))
|
||||
self.assertTrue(isinstance(runner, TestTaskRunner))
|
||||
|
|
|
|||
|
|
@ -8,8 +8,3 @@ class DevTests(unittest.TestCase, TestCommandMixin):
|
|||
result = self.invoke(["dev", "--help"])
|
||||
self.assertEqual(0, result.exit_code)
|
||||
self.assertIsNone(result.exception)
|
||||
|
||||
def test_dev_bindmount(self) -> None:
|
||||
result = self.invoke(["dev", "bindmount", "--help"])
|
||||
self.assertEqual(0, result.exit_code)
|
||||
self.assertIsNone(result.exception)
|
||||
|
|
|
|||
|
|
@ -0,0 +1,67 @@
|
|||
from unittest.mock import patch
|
||||
|
||||
from tests.helpers import PluginsTestCase, temporary_root
|
||||
from tutor.commands import jobs
|
||||
|
||||
from .base import TestCommandMixin
|
||||
|
||||
|
||||
class JobsTests(PluginsTestCase, TestCommandMixin):
|
||||
def test_initialise(self) -> None:
|
||||
with temporary_root() as root:
|
||||
self.invoke_in_root(root, ["config", "save"])
|
||||
result = self.invoke_in_root(root, ["local", "do", "init"])
|
||||
self.assertIsNone(result.exception)
|
||||
self.assertEqual(0, result.exit_code)
|
||||
self.assertIn("All services initialised.", result.output)
|
||||
|
||||
def test_create_user_template_without_staff(self) -> None:
|
||||
command = jobs.create_user_template(
|
||||
"superuser", False, "username", "email", "p4ssw0rd"
|
||||
)
|
||||
self.assertNotIn("--staff", command)
|
||||
self.assertIn("set_password", command)
|
||||
|
||||
def test_create_user_template_with_staff(self) -> None:
|
||||
command = jobs.create_user_template(
|
||||
"superuser", True, "username", "email", "p4ssw0rd"
|
||||
)
|
||||
self.assertIn("--staff", command)
|
||||
|
||||
def test_import_demo_course(self) -> None:
|
||||
with temporary_root() as root:
|
||||
self.invoke_in_root(root, ["config", "save"])
|
||||
with patch("tutor.utils.docker_compose") as mock_docker_compose:
|
||||
result = self.invoke_in_root(root, ["local", "do", "importdemocourse"])
|
||||
dc_args, _dc_kwargs = mock_docker_compose.call_args
|
||||
self.assertIsNone(result.exception)
|
||||
self.assertEqual(0, result.exit_code)
|
||||
self.assertIn("cms-job", dc_args)
|
||||
self.assertIn(
|
||||
"git clone https://github.com/openedx/edx-demo-course", dc_args[-1]
|
||||
)
|
||||
|
||||
def test_set_theme(self) -> None:
|
||||
with temporary_root() as root:
|
||||
self.invoke_in_root(root, ["config", "save"])
|
||||
with patch("tutor.utils.docker_compose") as mock_docker_compose:
|
||||
result = self.invoke_in_root(
|
||||
root,
|
||||
[
|
||||
"local",
|
||||
"do",
|
||||
"settheme",
|
||||
"--domain",
|
||||
"domain1",
|
||||
"--domain",
|
||||
"domain2",
|
||||
"beautiful",
|
||||
],
|
||||
)
|
||||
dc_args, _dc_kwargs = mock_docker_compose.call_args
|
||||
|
||||
self.assertIsNone(result.exception)
|
||||
self.assertEqual(0, result.exit_code)
|
||||
self.assertIn("lms-job", dc_args)
|
||||
self.assertIn("assign_theme('beautiful', 'domain1')", dc_args[-1])
|
||||
self.assertIn("assign_theme('beautiful', 'domain2')", dc_args[-1])
|
||||
|
|
@ -15,8 +15,8 @@ class LocalTests(unittest.TestCase, TestCommandMixin):
|
|||
self.assertIsNone(result.exception)
|
||||
self.assertEqual(0, result.exit_code)
|
||||
|
||||
def test_local_quickstart_help(self) -> None:
|
||||
result = self.invoke(["local", "quickstart", "--help"])
|
||||
def test_local_launch_help(self) -> None:
|
||||
result = self.invoke(["local", "launch", "--help"])
|
||||
self.assertIsNone(result.exception)
|
||||
self.assertEqual(0, result.exit_code)
|
||||
|
||||
|
|
|
|||
|
|
@ -5,12 +5,12 @@ import unittest
|
|||
import unittest.result
|
||||
|
||||
from tutor import hooks
|
||||
from tutor.commands.context import BaseJobContext
|
||||
from tutor.jobs import BaseJobRunner
|
||||
from tutor.commands.context import BaseTaskContext
|
||||
from tutor.tasks import BaseTaskRunner
|
||||
from tutor.types import Config
|
||||
|
||||
|
||||
class TestJobRunner(BaseJobRunner):
|
||||
class TestTaskRunner(BaseTaskRunner):
|
||||
"""
|
||||
Mock job runner for unit testing.
|
||||
|
||||
|
|
@ -18,7 +18,7 @@ class TestJobRunner(BaseJobRunner):
|
|||
separated by dashes.
|
||||
"""
|
||||
|
||||
def run_job(self, service: str, command: str) -> int:
|
||||
def run_task(self, service: str, command: str) -> int:
|
||||
print(os.linesep.join([f"Service: {service}", "-----", command, "----- "]))
|
||||
return 0
|
||||
|
||||
|
|
@ -36,13 +36,13 @@ def temporary_root() -> "tempfile.TemporaryDirectory[str]":
|
|||
return tempfile.TemporaryDirectory(prefix="tutor-test-root-")
|
||||
|
||||
|
||||
class TestContext(BaseJobContext):
|
||||
class TestContext(BaseTaskContext):
|
||||
"""
|
||||
Click context that will use only test job runners.
|
||||
"""
|
||||
|
||||
def job_runner(self, config: Config) -> TestJobRunner:
|
||||
return TestJobRunner(self.root, config)
|
||||
def job_runner(self, config: Config) -> TestTaskRunner:
|
||||
return TestTaskRunner(self.root, config)
|
||||
|
||||
|
||||
class PluginsTestCase(unittest.TestCase):
|
||||
|
|
|
|||
|
|
@ -1,36 +0,0 @@
|
|||
import unittest
|
||||
|
||||
from tutor import bindmounts
|
||||
from tutor.exceptions import TutorError
|
||||
|
||||
|
||||
class BindMountsTests(unittest.TestCase):
|
||||
def test_get_name(self) -> None:
|
||||
self.assertEqual("venv", bindmounts.get_name("/openedx/venv"))
|
||||
self.assertEqual("venv", bindmounts.get_name("/openedx/venv/"))
|
||||
|
||||
def test_get_name_root_folder(self) -> None:
|
||||
with self.assertRaises(TutorError):
|
||||
bindmounts.get_name("/")
|
||||
with self.assertRaises(TutorError):
|
||||
bindmounts.get_name("")
|
||||
|
||||
def test_parse_volumes(self) -> None:
|
||||
volume_args, non_volume_args = bindmounts.parse_volumes(
|
||||
[
|
||||
"run",
|
||||
"--volume=/openedx/venv",
|
||||
"-v",
|
||||
"/tmp/openedx:/openedx",
|
||||
"lms",
|
||||
"echo",
|
||||
"boom",
|
||||
]
|
||||
)
|
||||
self.assertEqual(("/openedx/venv", "/tmp/openedx:/openedx"), volume_args)
|
||||
self.assertEqual(("run", "lms", "echo", "boom"), non_volume_args)
|
||||
|
||||
def test_parse_volumes_empty_list(self) -> None:
|
||||
volume_args, non_volume_args = bindmounts.parse_volumes([])
|
||||
self.assertEqual((), volume_args)
|
||||
self.assertEqual((), non_volume_args)
|
||||
|
|
@ -42,8 +42,8 @@ class EnvTests(PluginsTestCase):
|
|||
self.assertTrue(env.is_binary_file("/home/somefile.ico"))
|
||||
|
||||
def test_find_os_path(self) -> None:
|
||||
renderer = env.Renderer()
|
||||
path = renderer.find_os_path("local/docker-compose.yml")
|
||||
environment = env.JinjaEnvironment()
|
||||
path = environment.find_os_path("local/docker-compose.yml")
|
||||
self.assertTrue(os.path.exists(path))
|
||||
|
||||
def test_pathjoin(self) -> None:
|
||||
|
|
@ -83,7 +83,7 @@ class EnvTests(PluginsTestCase):
|
|||
tutor_config.render_full(config)
|
||||
|
||||
config["MYSQL_ROOT_PASSWORD"] = "testpassword"
|
||||
rendered = env.render_file(config, "hooks", "mysql", "init")
|
||||
rendered = env.render_file(config, "jobs", "init", "mysql.sh")
|
||||
self.assertIn("testpassword", rendered)
|
||||
|
||||
@patch.object(fmt, "echo")
|
||||
|
|
|
|||
|
|
@ -1,82 +0,0 @@
|
|||
import re
|
||||
import unittest
|
||||
from io import StringIO
|
||||
from unittest.mock import patch
|
||||
|
||||
from tests.helpers import TestContext, temporary_root
|
||||
from tutor import config as tutor_config
|
||||
from tutor import jobs
|
||||
|
||||
|
||||
class JobsTests(unittest.TestCase):
|
||||
@patch("sys.stdout", new_callable=StringIO)
|
||||
def test_initialise(self, mock_stdout: StringIO) -> None:
|
||||
with temporary_root() as root:
|
||||
context = TestContext(root)
|
||||
config = tutor_config.load_full(root)
|
||||
runner = context.job_runner(config)
|
||||
jobs.initialise(runner)
|
||||
output = mock_stdout.getvalue().strip()
|
||||
self.assertTrue(output.startswith("Initialising all services..."))
|
||||
self.assertTrue(output.endswith("All services initialised."))
|
||||
|
||||
def test_create_user_command_without_staff(self) -> None:
|
||||
command = jobs.create_user_command("superuser", False, "username", "email")
|
||||
self.assertNotIn("--staff", command)
|
||||
|
||||
def test_create_user_command_with_staff(self) -> None:
|
||||
command = jobs.create_user_command("superuser", True, "username", "email")
|
||||
self.assertIn("--staff", command)
|
||||
|
||||
def test_create_user_command_with_staff_with_password(self) -> None:
|
||||
command = jobs.create_user_command(
|
||||
"superuser", True, "username", "email", "command"
|
||||
)
|
||||
self.assertIn("set_password", command)
|
||||
|
||||
@patch("sys.stdout", new_callable=StringIO)
|
||||
def test_import_demo_course(self, mock_stdout: StringIO) -> None:
|
||||
with temporary_root() as root:
|
||||
context = TestContext(root)
|
||||
config = tutor_config.load_full(root)
|
||||
runner = context.job_runner(config)
|
||||
jobs.import_demo_course(runner)
|
||||
|
||||
output = mock_stdout.getvalue()
|
||||
service = re.search(r"Service: (\w*)", output)
|
||||
commands = re.search(r"(-----)([\S\s]+)(-----)", output)
|
||||
assert service is not None
|
||||
assert commands is not None
|
||||
self.assertEqual(service.group(1), "cms")
|
||||
self.assertTrue(
|
||||
commands.group(2)
|
||||
.strip()
|
||||
.startswith('echo "Loading settings $DJANGO_SETTINGS_MODULE"')
|
||||
)
|
||||
|
||||
@patch("sys.stdout", new_callable=StringIO)
|
||||
def test_set_theme(self, mock_stdout: StringIO) -> None:
|
||||
with temporary_root() as root:
|
||||
context = TestContext(root)
|
||||
config = tutor_config.load_full(root)
|
||||
runner = context.job_runner(config)
|
||||
jobs.set_theme("sample_theme", ["domain1", "domain2"], runner)
|
||||
|
||||
output = mock_stdout.getvalue()
|
||||
service = re.search(r"Service: (\w*)", output)
|
||||
commands = re.search(r"(-----)([\S\s]+)(-----)", output)
|
||||
assert service is not None
|
||||
assert commands is not None
|
||||
self.assertEqual(service.group(1), "lms")
|
||||
self.assertTrue(
|
||||
commands.group(2)
|
||||
.strip()
|
||||
.startswith('echo "Loading settings $DJANGO_SETTINGS_MODULE"')
|
||||
)
|
||||
|
||||
def test_get_all_openedx_domains(self) -> None:
|
||||
with temporary_root() as root:
|
||||
config = tutor_config.load_full(root)
|
||||
domains = jobs.get_all_openedx_domains(config)
|
||||
self.assertTrue(domains)
|
||||
self.assertEqual(6, len(domains))
|
||||
|
|
@ -167,10 +167,17 @@ class PluginsTests(PluginsTestCase):
|
|||
|
||||
def test_init_tasks(self) -> None:
|
||||
plugins_v0.DictPlugin({"name": "plugin1", "hooks": {"init": ["myclient"]}})
|
||||
plugins.load("plugin1")
|
||||
with patch.object(
|
||||
plugins_v0.env, "read_template_file", return_value="echo hello"
|
||||
) as mock_read_template:
|
||||
plugins.load("plugin1")
|
||||
mock_read_template.assert_called_once_with(
|
||||
"plugin1", "hooks", "myclient", "init"
|
||||
)
|
||||
|
||||
self.assertIn(
|
||||
("myclient", ("plugin1", "hooks", "myclient", "init")),
|
||||
list(hooks.Filters.COMMANDS_INIT.iterate()),
|
||||
("myclient", "echo hello"),
|
||||
list(hooks.Filters.CLI_DO_INIT_TASKS.iterate()),
|
||||
)
|
||||
|
||||
def test_plugins_are_updated_on_config_change(self) -> None:
|
||||
|
|
|
|||
|
|
@ -10,7 +10,7 @@ __version__ = "14.2.3"
|
|||
# the nightly branch.
|
||||
# The suffix is cleanly separated from the __version__ in this module to avoid
|
||||
# conflicts when merging branches.
|
||||
__version_suffix__ = ""
|
||||
__version_suffix__ = "nightly"
|
||||
|
||||
# The app name will be used to define the name of the default tutor root and
|
||||
# plugin directory. To avoid conflicts between multiple locally-installed
|
||||
|
|
|
|||
|
|
@ -1,15 +1,12 @@
|
|||
import os
|
||||
from typing import List, Tuple
|
||||
|
||||
import click
|
||||
|
||||
from .exceptions import TutorError
|
||||
from .jobs import BaseComposeJobRunner
|
||||
from .utils import get_user_id
|
||||
from tutor.exceptions import TutorError
|
||||
from tutor.tasks import BaseComposeTaskRunner
|
||||
from tutor.utils import get_user_id
|
||||
|
||||
|
||||
def create(
|
||||
runner: BaseComposeJobRunner,
|
||||
runner: BaseComposeTaskRunner,
|
||||
service: str,
|
||||
path: str,
|
||||
) -> str:
|
||||
|
|
@ -62,22 +59,3 @@ def get_name(container_bind_path: str) -> str:
|
|||
|
||||
def get_root_path(root: str) -> str:
|
||||
return os.path.join(root, "volumes")
|
||||
|
||||
|
||||
def parse_volumes(docker_compose_args: List[str]) -> Tuple[List[str], List[str]]:
|
||||
"""
|
||||
Parse `-v/--volume` options from an arbitrary list of arguments.
|
||||
"""
|
||||
|
||||
@click.command(context_settings={"ignore_unknown_options": True})
|
||||
@click.option("-v", "--volume", "volumes", multiple=True)
|
||||
@click.argument("args", nargs=-1)
|
||||
def custom_docker_compose(
|
||||
volumes: List[str], args: List[str] # pylint: disable=unused-argument
|
||||
) -> None:
|
||||
pass
|
||||
|
||||
if isinstance(docker_compose_args, tuple):
|
||||
docker_compose_args = list(docker_compose_args)
|
||||
context = custom_docker_compose.make_context("custom", docker_compose_args)
|
||||
return context.params["volumes"], context.params["args"]
|
||||
|
|
|
|||
|
|
@ -115,6 +115,7 @@ def cli(context: click.Context, root: str, show_help: bool) -> None:
|
|||
"/install/linux/linux-postinstall/#manage-docker-as-a-non-root-user)"
|
||||
)
|
||||
context.obj = Context(root)
|
||||
context.help_option_names = ["-h", "--help"]
|
||||
if context.invoked_subcommand is None or show_help:
|
||||
click.echo(context.get_help())
|
||||
|
||||
|
|
|
|||
|
|
@ -7,18 +7,19 @@ import click
|
|||
from click.shell_completion import CompletionItem
|
||||
from typing_extensions import TypeAlias
|
||||
|
||||
from tutor import bindmounts
|
||||
from tutor import config as tutor_config
|
||||
from tutor import env as tutor_env
|
||||
from tutor import fmt, hooks, jobs, serialize, utils
|
||||
from tutor.commands.context import BaseJobContext
|
||||
from tutor import fmt, hooks, serialize, utils
|
||||
from tutor.commands import jobs
|
||||
from tutor.commands.context import BaseTaskContext
|
||||
from tutor.exceptions import TutorError
|
||||
from tutor.tasks import BaseComposeTaskRunner
|
||||
from tutor.types import Config
|
||||
|
||||
COMPOSE_FILTER_TYPE: TypeAlias = "hooks.filters.Filter[t.Dict[str, t.Any], []]"
|
||||
|
||||
|
||||
class ComposeJobRunner(jobs.BaseComposeJobRunner):
|
||||
class ComposeTaskRunner(BaseComposeTaskRunner):
|
||||
def __init__(self, root: str, config: Config):
|
||||
super().__init__(root, config)
|
||||
self.project_name = ""
|
||||
|
|
@ -84,7 +85,7 @@ class ComposeJobRunner(jobs.BaseComposeJobRunner):
|
|||
docker_compose_jobs_tmp_path,
|
||||
)
|
||||
|
||||
def run_job(self, service: str, command: str) -> int:
|
||||
def run_task(self, service: str, command: str) -> int:
|
||||
"""
|
||||
Run the "{{ service }}-job" service from local/docker-compose.jobs.yml with the
|
||||
specified command.
|
||||
|
|
@ -108,11 +109,11 @@ class ComposeJobRunner(jobs.BaseComposeJobRunner):
|
|||
)
|
||||
|
||||
|
||||
class BaseComposeContext(BaseJobContext):
|
||||
class BaseComposeContext(BaseTaskContext):
|
||||
COMPOSE_TMP_FILTER: COMPOSE_FILTER_TYPE = NotImplemented
|
||||
COMPOSE_JOBS_TMP_FILTER: COMPOSE_FILTER_TYPE = NotImplemented
|
||||
|
||||
def job_runner(self, config: Config) -> ComposeJobRunner:
|
||||
def job_runner(self, config: Config) -> ComposeTaskRunner:
|
||||
raise NotImplementedError
|
||||
|
||||
|
||||
|
|
@ -311,77 +312,23 @@ def restart(context: BaseComposeContext, services: t.List[str]) -> None:
|
|||
context.job_runner(config).docker_compose(*command)
|
||||
|
||||
|
||||
@click.command(help="Initialise all applications")
|
||||
@click.option("-l", "--limit", help="Limit initialisation to this service or plugin")
|
||||
@jobs.do_group
|
||||
@mount_option
|
||||
@click.pass_obj
|
||||
def init(
|
||||
context: BaseComposeContext,
|
||||
limit: str,
|
||||
mounts: t.Tuple[t.List[MountParam.MountType]],
|
||||
def do(
|
||||
context: BaseComposeContext, mounts: t.Tuple[t.List[MountParam.MountType]]
|
||||
) -> None:
|
||||
mount_tmp_volumes(mounts, context)
|
||||
config = tutor_config.load(context.root)
|
||||
runner = context.job_runner(config)
|
||||
jobs.initialise(runner, limit_to=limit)
|
||||
"""
|
||||
Run a custom job in the right container(s).
|
||||
"""
|
||||
|
||||
|
||||
@click.command(help="Create an Open edX user and interactively set their password")
|
||||
@click.option("--superuser", is_flag=True, help="Make superuser")
|
||||
@click.option("--staff", is_flag=True, help="Make staff user")
|
||||
@click.option(
|
||||
"-p",
|
||||
"--password",
|
||||
help="Specify password from the command line. If undefined, you will be prompted to input a password",
|
||||
)
|
||||
@click.argument("name")
|
||||
@click.argument("email")
|
||||
@click.pass_obj
|
||||
def createuser(
|
||||
context: BaseComposeContext,
|
||||
superuser: str,
|
||||
staff: bool,
|
||||
password: str,
|
||||
name: str,
|
||||
email: str,
|
||||
) -> None:
|
||||
config = tutor_config.load(context.root)
|
||||
runner = context.job_runner(config)
|
||||
command = jobs.create_user_command(superuser, staff, name, email, password=password)
|
||||
runner.run_job("lms", command)
|
||||
|
||||
|
||||
@click.command(
|
||||
help="Assign a theme to the LMS and the CMS. To reset to the default theme , use 'default' as the theme name."
|
||||
)
|
||||
@click.option(
|
||||
"-d",
|
||||
"--domain",
|
||||
"domains",
|
||||
multiple=True,
|
||||
help=(
|
||||
"Limit the theme to these domain names. By default, the theme is "
|
||||
"applied to the LMS and the CMS, both in development and production mode"
|
||||
),
|
||||
)
|
||||
@click.argument("theme_name")
|
||||
@click.pass_obj
|
||||
def settheme(
|
||||
context: BaseComposeContext, domains: t.List[str], theme_name: str
|
||||
) -> None:
|
||||
config = tutor_config.load(context.root)
|
||||
runner = context.job_runner(config)
|
||||
domains = domains or jobs.get_all_openedx_domains(config)
|
||||
jobs.set_theme(theme_name, domains, runner)
|
||||
|
||||
|
||||
@click.command(help="Import the demo course")
|
||||
@click.pass_obj
|
||||
def importdemocourse(context: BaseComposeContext) -> None:
|
||||
config = tutor_config.load(context.root)
|
||||
runner = context.job_runner(config)
|
||||
fmt.echo_info("Importing demo course")
|
||||
jobs.import_demo_course(runner)
|
||||
@hooks.Actions.DO_JOB.add()
|
||||
def _mount_tmp_volumes(_job_name: str, *_args: t.Any, **_kwargs: t.Any) -> None:
|
||||
"""
|
||||
We add this logic to an action callback because we do not want to trigger it
|
||||
whenever we run `tutor local do <job> --help`.
|
||||
"""
|
||||
mount_tmp_volumes(mounts, context)
|
||||
|
||||
|
||||
@click.command(
|
||||
|
|
@ -407,28 +354,6 @@ def run(
|
|||
context.invoke(dc_command, mounts=mounts, command="run", args=[*extra_args, *args])
|
||||
|
||||
|
||||
@click.command(
|
||||
name="bindmount",
|
||||
help="Copy the contents of a container directory to a ready-to-bind-mount host directory",
|
||||
)
|
||||
@click.argument("service")
|
||||
@click.argument("path")
|
||||
@click.pass_obj
|
||||
def bindmount_command(context: BaseComposeContext, service: str, path: str) -> None:
|
||||
"""
|
||||
This command is made obsolete by the --mount arguments.
|
||||
"""
|
||||
fmt.echo_alert(
|
||||
"The 'bindmount' command is deprecated and will be removed in a later release. Use 'copyfrom' instead."
|
||||
)
|
||||
config = tutor_config.load(context.root)
|
||||
host_path = bindmounts.create(context.job_runner(config), service, path)
|
||||
fmt.echo_info(
|
||||
f"Bind-mount volume created at {host_path}. You can now use it in all `local` and `dev` "
|
||||
f"commands with the `--volume={path}` option."
|
||||
)
|
||||
|
||||
|
||||
@click.command(
|
||||
name="copyfrom",
|
||||
help="Copy files/folders from a container directory to the local filesystem.",
|
||||
|
|
@ -535,20 +460,7 @@ def dc_command(
|
|||
) -> None:
|
||||
mount_tmp_volumes(mounts, context)
|
||||
config = tutor_config.load(context.root)
|
||||
volumes, non_volume_args = bindmounts.parse_volumes(args)
|
||||
volume_args = []
|
||||
for volume_arg in volumes:
|
||||
if ":" not in volume_arg:
|
||||
# This is a bind-mounted volume from the "volumes/" folder.
|
||||
host_bind_path = bindmounts.get_path(context.root, volume_arg)
|
||||
if not os.path.exists(host_bind_path):
|
||||
raise TutorError(
|
||||
f"Bind-mount volume directory {host_bind_path} does not exist. It must first be created "
|
||||
f"with the '{bindmount_command.name}' command."
|
||||
)
|
||||
volume_arg = f"{host_bind_path}:{volume_arg}"
|
||||
volume_args += ["--volume", volume_arg]
|
||||
context.job_runner(config).docker_compose(command, *volume_args, *non_volume_args)
|
||||
context.job_runner(config).docker_compose(command, *args)
|
||||
|
||||
|
||||
@hooks.Filters.COMPOSE_MOUNTS.add()
|
||||
|
|
@ -577,14 +489,14 @@ def add_commands(command_group: click.Group) -> None:
|
|||
command_group.add_command(stop)
|
||||
command_group.add_command(restart)
|
||||
command_group.add_command(reboot)
|
||||
command_group.add_command(init)
|
||||
command_group.add_command(createuser)
|
||||
command_group.add_command(importdemocourse)
|
||||
command_group.add_command(settheme)
|
||||
command_group.add_command(dc_command)
|
||||
command_group.add_command(run)
|
||||
command_group.add_command(copyfrom)
|
||||
command_group.add_command(bindmount_command)
|
||||
command_group.add_command(execute)
|
||||
command_group.add_command(logs)
|
||||
command_group.add_command(status)
|
||||
|
||||
@hooks.Actions.PLUGINS_LOADED.add()
|
||||
def _add_do_commands() -> None:
|
||||
jobs.add_job_commands(do)
|
||||
command_group.add_command(do)
|
||||
|
|
|
|||
|
|
@ -1,5 +1,5 @@
|
|||
from ..jobs import BaseJobRunner
|
||||
from ..types import Config
|
||||
from tutor.tasks import BaseTaskRunner
|
||||
from tutor.types import Config
|
||||
|
||||
|
||||
class Context:
|
||||
|
|
@ -16,14 +16,14 @@ class Context:
|
|||
self.root = root
|
||||
|
||||
|
||||
class BaseJobContext(Context):
|
||||
class BaseTaskContext(Context):
|
||||
"""
|
||||
Specialized context that subcommands may use.
|
||||
|
||||
For instance `dev`, `local` and `k8s` define custom runners to run jobs.
|
||||
"""
|
||||
|
||||
def job_runner(self, config: Config) -> BaseJobRunner:
|
||||
def job_runner(self, config: Config) -> BaseTaskRunner:
|
||||
"""
|
||||
Return a runner capable of running docker-compose/kubectl commands.
|
||||
"""
|
||||
|
|
|
|||
|
|
@ -11,7 +11,7 @@ from tutor.commands import compose
|
|||
from tutor.types import Config, get_typed
|
||||
|
||||
|
||||
class DevJobRunner(compose.ComposeJobRunner):
|
||||
class DevTaskRunner(compose.ComposeTaskRunner):
|
||||
def __init__(self, root: str, config: Config):
|
||||
"""
|
||||
Load docker-compose files from dev/ and local/
|
||||
|
|
@ -51,8 +51,8 @@ class DevContext(compose.BaseComposeContext):
|
|||
COMPOSE_TMP_FILTER = hooks.Filters.COMPOSE_DEV_TMP
|
||||
COMPOSE_JOBS_TMP_FILTER = hooks.Filters.COMPOSE_DEV_JOBS_TMP
|
||||
|
||||
def job_runner(self, config: Config) -> DevJobRunner:
|
||||
return DevJobRunner(self.root, config)
|
||||
def job_runner(self, config: Config) -> DevTaskRunner:
|
||||
return DevTaskRunner(self.root, config)
|
||||
|
||||
|
||||
@click.group(help="Run Open edX locally with development settings")
|
||||
|
|
@ -66,7 +66,7 @@ def dev(context: click.Context) -> None:
|
|||
@click.option("-p", "--pullimages", is_flag=True, help="Update docker images")
|
||||
@compose.mount_option
|
||||
@click.pass_context
|
||||
def quickstart(
|
||||
def launch(
|
||||
context: click.Context,
|
||||
non_interactive: bool,
|
||||
pullimages: bool,
|
||||
|
|
@ -105,7 +105,7 @@ Tutor may not work if Docker is configured with < 4 GB RAM. Please follow instru
|
|||
context.invoke(compose.start, detach=True)
|
||||
|
||||
click.echo(fmt.title("Database creation and migrations"))
|
||||
context.invoke(compose.init)
|
||||
context.invoke(compose.do.commands["init"])
|
||||
|
||||
fmt.echo_info(
|
||||
"""The Open edX platform is now running in detached mode
|
||||
|
|
@ -120,35 +120,26 @@ Your Open edX platform is ready and can be accessed at the following urls:
|
|||
)
|
||||
|
||||
|
||||
@click.command(
|
||||
help="DEPRECATED: Use 'tutor dev start ...' instead!",
|
||||
context_settings={"ignore_unknown_options": True},
|
||||
)
|
||||
@click.command(help="Deprecated alias to 'launch'")
|
||||
@click.option("-I", "--non-interactive", is_flag=True, help="Run non-interactively")
|
||||
@click.option("-p", "--pullimages", is_flag=True, help="Update docker images")
|
||||
@compose.mount_option
|
||||
@click.argument("options", nargs=-1, required=False)
|
||||
@click.argument("service")
|
||||
@click.pass_context
|
||||
def runserver(
|
||||
def quickstart(
|
||||
context: click.Context,
|
||||
non_interactive: bool,
|
||||
pullimages: bool,
|
||||
mounts: t.Tuple[t.List[compose.MountParam.MountType]],
|
||||
options: t.List[str],
|
||||
service: str,
|
||||
) -> None:
|
||||
depr_warning = "'runserver' is deprecated and will be removed in a future release. Use 'start' instead."
|
||||
for option in options:
|
||||
if option.startswith("-v") or option.startswith("--volume"):
|
||||
depr_warning += " Bind-mounts can be specified using '-m/--mount'."
|
||||
break
|
||||
fmt.echo_alert(depr_warning)
|
||||
config = tutor_config.load(context.obj.root)
|
||||
if service in ["lms", "cms"]:
|
||||
port = 8000 if service == "lms" else 8001
|
||||
host = config["LMS_HOST"] if service == "lms" else config["CMS_HOST"]
|
||||
fmt.echo_info(
|
||||
f"The {service} service will be available at http://{host}:{port}"
|
||||
)
|
||||
args = ["--service-ports", *options, service]
|
||||
context.invoke(compose.run, mounts=mounts, args=args)
|
||||
"""
|
||||
This command has been renamed to 'launch'.
|
||||
"""
|
||||
fmt.echo_alert(
|
||||
"The 'quickstart' command is deprecated and will be removed in a later release. Use 'launch' instead."
|
||||
)
|
||||
context.invoke(
|
||||
launch, non_interactive=non_interactive, pullimages=pullimages, mounts=mounts
|
||||
)
|
||||
|
||||
|
||||
@hooks.Actions.COMPOSE_PROJECT_STARTED.add()
|
||||
|
|
@ -157,11 +148,11 @@ def _stop_on_local_start(root: str, config: Config, project_name: str) -> None:
|
|||
Stop the dev platform as soon as a platform with a different project name is
|
||||
started.
|
||||
"""
|
||||
runner = DevJobRunner(root, config)
|
||||
runner = DevTaskRunner(root, config)
|
||||
if project_name != runner.project_name:
|
||||
runner.docker_compose("stop")
|
||||
|
||||
|
||||
dev.add_command(launch)
|
||||
dev.add_command(quickstart)
|
||||
dev.add_command(runserver)
|
||||
compose.add_commands(dev)
|
||||
|
|
|
|||
|
|
@ -0,0 +1,278 @@
|
|||
"""
|
||||
Common jobs that must be added both to local, dev and k8s commands.
|
||||
"""
|
||||
import functools
|
||||
import typing as t
|
||||
|
||||
import click
|
||||
from typing_extensions import ParamSpec
|
||||
|
||||
from tutor import config as tutor_config
|
||||
from tutor import env, fmt, hooks
|
||||
|
||||
|
||||
class DoGroup(click.Group):
|
||||
"""
|
||||
A Click group that prints subcommands under 'Jobs' instead of 'Commands' when we run
|
||||
`.. do --help`. Hackish but it works.
|
||||
"""
|
||||
|
||||
def get_help(self, ctx: click.Context) -> str:
|
||||
return super().get_help(ctx).replace("Commands:\n", "Jobs:\n")
|
||||
|
||||
|
||||
# A convenient easy-to-use decorator for creating `do` commands.
|
||||
do_group = click.group(cls=DoGroup, subcommand_metavar="JOB [ARGS]...")
|
||||
|
||||
|
||||
@hooks.Actions.CORE_READY.add()
|
||||
def _add_core_init_tasks() -> None:
|
||||
"""
|
||||
Declare core init scripts at runtime.
|
||||
|
||||
The context is important, because it allows us to select the init scripts based on
|
||||
the --limit argument.
|
||||
"""
|
||||
with hooks.Contexts.APP("mysql").enter():
|
||||
hooks.Filters.CLI_DO_INIT_TASKS.add_item(
|
||||
("mysql", env.read_core_template_file("jobs", "init", "mysql.sh"))
|
||||
)
|
||||
with hooks.Contexts.APP("lms").enter():
|
||||
hooks.Filters.CLI_DO_INIT_TASKS.add_item(
|
||||
("lms", env.read_core_template_file("jobs", "init", "lms.sh"))
|
||||
)
|
||||
with hooks.Contexts.APP("cms").enter():
|
||||
hooks.Filters.CLI_DO_INIT_TASKS.add_item(
|
||||
("cms", env.read_core_template_file("jobs", "init", "cms.sh"))
|
||||
)
|
||||
|
||||
|
||||
@click.command("init", help="Initialise all applications")
|
||||
@click.option("-l", "--limit", help="Limit initialisation to this service or plugin")
|
||||
def initialise(limit: t.Optional[str]) -> t.Iterator[t.Tuple[str, str]]:
|
||||
fmt.echo_info("Initialising all services...")
|
||||
filter_context = hooks.Contexts.APP(limit).name if limit else None
|
||||
|
||||
# Deprecated pre-init tasks
|
||||
for service, path in hooks.Filters.COMMANDS_PRE_INIT.iterate_from_context(
|
||||
filter_context
|
||||
):
|
||||
fmt.echo_alert(
|
||||
f"Running deprecated pre-init task: {'/'.join(path)}. Init tasks should no longer be added to the COMMANDS_PRE_INIT filter. Plugin developers should use the CLI_DO_INIT_TASKS filter instead, with a high priority."
|
||||
)
|
||||
yield service, env.read_template_file(*path)
|
||||
|
||||
# Init tasks
|
||||
for service, task in hooks.Filters.CLI_DO_INIT_TASKS.iterate_from_context(
|
||||
filter_context
|
||||
):
|
||||
fmt.echo_info(f"Running init task in {service}")
|
||||
yield service, task
|
||||
|
||||
# Deprecated init tasks
|
||||
for service, path in hooks.Filters.COMMANDS_INIT.iterate_from_context(
|
||||
filter_context
|
||||
):
|
||||
fmt.echo_alert(
|
||||
f"Running deprecated init task: {'/'.join(path)}. Init tasks should no longer be added to the COMMANDS_INIT filter. Plugin developers should use the CLI_DO_INIT_TASKS filter instead."
|
||||
)
|
||||
yield service, env.read_template_file(*path)
|
||||
|
||||
fmt.echo_info("All services initialised.")
|
||||
|
||||
|
||||
@click.command(help="Create an Open edX user and interactively set their password")
|
||||
@click.option("--superuser", is_flag=True, help="Make superuser")
|
||||
@click.option("--staff", is_flag=True, help="Make staff user")
|
||||
@click.option(
|
||||
"-p",
|
||||
"--password",
|
||||
help="Specify password from the command line. If undefined, you will be prompted to input a password",
|
||||
prompt=True,
|
||||
hide_input=True,
|
||||
)
|
||||
@click.argument("name")
|
||||
@click.argument("email")
|
||||
def createuser(
|
||||
superuser: str,
|
||||
staff: bool,
|
||||
password: str,
|
||||
name: str,
|
||||
email: str,
|
||||
) -> t.Iterable[t.Tuple[str, str]]:
|
||||
"""
|
||||
Create an Open edX user
|
||||
|
||||
Password can be passed as an option or will be set interactively.
|
||||
"""
|
||||
yield ("lms", create_user_template(superuser, staff, name, email, password))
|
||||
|
||||
|
||||
def create_user_template(
|
||||
superuser: str, staff: bool, username: str, email: str, password: str
|
||||
) -> str:
|
||||
opts = ""
|
||||
if superuser:
|
||||
opts += " --superuser"
|
||||
if staff:
|
||||
opts += " --staff"
|
||||
return f"""
|
||||
./manage.py lms manage_user {opts} {username} {email}
|
||||
./manage.py lms shell -c "
|
||||
from django.contrib.auth import get_user_model
|
||||
u = get_user_model().objects.get(username='{username}')
|
||||
u.set_password('{password}')
|
||||
u.save()"
|
||||
"""
|
||||
|
||||
|
||||
@click.command(help="Import the demo course")
|
||||
def importdemocourse() -> t.Iterable[t.Tuple[str, str]]:
|
||||
template = """
|
||||
# Import demo course
|
||||
git clone https://github.com/openedx/edx-demo-course --branch {{ OPENEDX_COMMON_VERSION }} --depth 1 ../edx-demo-course
|
||||
python ./manage.py cms import ../data ../edx-demo-course
|
||||
|
||||
# Re-index courses
|
||||
./manage.py cms reindex_course --all --setup"""
|
||||
yield ("cms", template)
|
||||
|
||||
|
||||
@click.command()
|
||||
@click.option(
|
||||
"-d",
|
||||
"--domain",
|
||||
"domains",
|
||||
multiple=True,
|
||||
help=(
|
||||
"Limit the theme to these domain names. By default, the theme is "
|
||||
"applied to the LMS and the CMS, both in development and production mode"
|
||||
),
|
||||
)
|
||||
@click.argument("theme_name")
|
||||
def settheme(domains: t.List[str], theme_name: str) -> t.Iterable[t.Tuple[str, str]]:
|
||||
"""
|
||||
Assign a theme to the LMS and the CMS.
|
||||
|
||||
To reset to the default theme , use 'default' as the theme name.
|
||||
"""
|
||||
yield ("lms", set_theme_template(theme_name, domains))
|
||||
|
||||
|
||||
def set_theme_template(theme_name: str, domain_names: t.List[str]) -> str:
|
||||
"""
|
||||
For each domain, get or create a Site object and assign the selected theme.
|
||||
"""
|
||||
# Note that there are no double quotes " in this piece of code
|
||||
python_command = """
|
||||
import sys
|
||||
from django.contrib.sites.models import Site
|
||||
def assign_theme(name, domain):
|
||||
print('Assigning theme', name, 'to', domain)
|
||||
if len(domain) > 50:
|
||||
sys.stderr.write(
|
||||
'Assigning a theme to a site with a long (> 50 characters) domain name.'
|
||||
' The displayed site name will be truncated to 50 characters.\\n'
|
||||
)
|
||||
site, _ = Site.objects.get_or_create(domain=domain)
|
||||
if not site.name:
|
||||
name_max_length = Site._meta.get_field('name').max_length
|
||||
site.name = domain[:name_max_length]
|
||||
site.save()
|
||||
site.themes.all().delete()
|
||||
site.themes.create(theme_dir_name=name)
|
||||
"""
|
||||
domain_names = domain_names or [
|
||||
"{{ LMS_HOST }}",
|
||||
"{{ LMS_HOST }}:8000",
|
||||
"{{ CMS_HOST }}",
|
||||
"{{ CMS_HOST }}:8001",
|
||||
"{{ PREVIEW_LMS_HOST }}",
|
||||
"{{ PREVIEW_LMS_HOST }}:8000",
|
||||
]
|
||||
for domain_name in domain_names:
|
||||
python_command += f"assign_theme('{theme_name}', '{domain_name}')\n"
|
||||
return f'./manage.py lms shell -c "{python_command}"'
|
||||
|
||||
|
||||
def add_job_commands(do_command_group: click.Group) -> None:
|
||||
"""
|
||||
This is meant to be called with the `local/dev/k8s do` group commands, to add the
|
||||
different `do` subcommands.
|
||||
"""
|
||||
for subcommand in hooks.Filters.CLI_DO_COMMANDS.iterate():
|
||||
assert isinstance(subcommand, click.Command)
|
||||
do_command_group.add_command(subcommand)
|
||||
|
||||
|
||||
@hooks.Actions.PLUGINS_LOADED.add()
|
||||
def _patch_do_commands_callbacks() -> None:
|
||||
"""
|
||||
After plugins have been loaded, patch `do` subcommands such that their output is
|
||||
forwarded to `do_callback`.
|
||||
|
||||
This function is not called as part of add_job_commands because subcommands must be
|
||||
patched just once.
|
||||
"""
|
||||
for subcommand in hooks.Filters.CLI_DO_COMMANDS.iterate():
|
||||
if not isinstance(subcommand, click.Command):
|
||||
raise ValueError(
|
||||
f"Command {subcommand} which was added to the CLI_DO_COMMANDS filter must be an instance of click.Command"
|
||||
)
|
||||
# Modify the subcommand callback such that job results are processed by do_callback
|
||||
if subcommand.callback is None:
|
||||
raise ValueError("Cannot patch None callback")
|
||||
if subcommand.name is None:
|
||||
raise ValueError("Defined job with None name")
|
||||
subcommand.callback = _patch_callback(subcommand.name, subcommand.callback)
|
||||
|
||||
|
||||
P = ParamSpec("P")
|
||||
|
||||
|
||||
def _patch_callback(
|
||||
job_name: str, func: t.Callable[P, t.Iterable[t.Tuple[str, str]]]
|
||||
) -> t.Callable[P, None]:
|
||||
"""
|
||||
Modify a subcommand callback function such that its results are processed by `do_callback`.
|
||||
"""
|
||||
|
||||
def new_callback(*args: P.args, **kwargs: P.kwargs) -> None:
|
||||
hooks.Actions.DO_JOB.do(job_name, *args, **kwargs)
|
||||
do_callback(func(*args, **kwargs))
|
||||
|
||||
# Make the new callback behave like the old one
|
||||
functools.update_wrapper(new_callback, func)
|
||||
|
||||
return new_callback
|
||||
|
||||
|
||||
def do_callback(service_commands: t.Iterable[t.Tuple[str, str]]) -> None:
|
||||
"""
|
||||
This function must be added as a callback to all `do` subcommands.
|
||||
|
||||
`do` subcommands don't actually run any task. They just yield tuples of (service
|
||||
name, unrendered script string). This function is responsible for actually running
|
||||
the scripts. It does the following:
|
||||
|
||||
- Prefix the script with a base command
|
||||
- Render the script string
|
||||
- Run a job in the right container
|
||||
|
||||
This callback is added to the "do" subcommands by the `add_job_commands` function.
|
||||
"""
|
||||
context = click.get_current_context().obj
|
||||
config = tutor_config.load(context.root)
|
||||
runner = context.job_runner(config)
|
||||
for service, command in service_commands:
|
||||
runner.run_task_from_str(service, command)
|
||||
|
||||
|
||||
hooks.Filters.CLI_DO_COMMANDS.add_items(
|
||||
[
|
||||
createuser,
|
||||
importdemocourse,
|
||||
initialise,
|
||||
settheme,
|
||||
]
|
||||
)
|
||||
|
|
@ -6,12 +6,14 @@ import click
|
|||
|
||||
from tutor import config as tutor_config
|
||||
from tutor import env as tutor_env
|
||||
from tutor import exceptions, fmt
|
||||
from tutor import exceptions, fmt, hooks
|
||||
from tutor import interactive as interactive_config
|
||||
from tutor import jobs, serialize, utils
|
||||
from tutor import serialize, utils
|
||||
from tutor.commands import jobs
|
||||
from tutor.commands.config import save as config_save_command
|
||||
from tutor.commands.context import BaseJobContext
|
||||
from tutor.commands.context import BaseTaskContext
|
||||
from tutor.commands.upgrade.k8s import upgrade_from
|
||||
from tutor.tasks import BaseTaskRunner
|
||||
from tutor.types import Config, get_typed
|
||||
|
||||
|
||||
|
|
@ -20,7 +22,8 @@ class K8sClients:
|
|||
|
||||
def __init__(self) -> None:
|
||||
# Loading the kubernetes module here to avoid import overhead
|
||||
from kubernetes import client, config # pylint: disable=import-outside-toplevel
|
||||
# pylint: disable=import-outside-toplevel
|
||||
from kubernetes import client, config
|
||||
|
||||
config.load_kube_config()
|
||||
self._batch_api = None
|
||||
|
|
@ -46,33 +49,20 @@ class K8sClients:
|
|||
return self._core_api
|
||||
|
||||
|
||||
class K8sJobRunner(jobs.BaseJobRunner):
|
||||
def load_job(self, name: str) -> Any:
|
||||
all_jobs = self.render("k8s", "jobs.yml")
|
||||
for job in serialize.load_all(all_jobs):
|
||||
job_name = job["metadata"]["name"]
|
||||
if not isinstance(job_name, str):
|
||||
raise exceptions.TutorError(
|
||||
f"Invalid job name: '{job_name}'. Expected str."
|
||||
)
|
||||
if job_name == name:
|
||||
return job
|
||||
raise exceptions.TutorError(f"Could not find job '{name}'")
|
||||
class K8sTaskRunner(BaseTaskRunner):
|
||||
"""
|
||||
Run tasks (bash commands) in Kubernetes-managed services.
|
||||
|
||||
def active_job_names(self) -> List[str]:
|
||||
"""
|
||||
Return a list of active job names
|
||||
Docs:
|
||||
https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.18/#list-job-v1-batch
|
||||
"""
|
||||
api = K8sClients.instance().batch_api
|
||||
return [
|
||||
job.metadata.name
|
||||
for job in api.list_namespaced_job(k8s_namespace(self.config)).items
|
||||
if job.status.active
|
||||
]
|
||||
Note: a single Tutor "task" correspond to a Kubernetes "job":
|
||||
https://kubernetes.io/docs/concepts/workloads/controllers/job/
|
||||
A Tutor "job" is composed of multiple Tutor tasks run in different services.
|
||||
|
||||
def run_job(self, service: str, command: str) -> int:
|
||||
In Kubernetes, each task that is expected to run in a "myservice" container will
|
||||
trigger the "myservice-job" Kubernetes job. This job definition must be present in
|
||||
the "k8s/jobs.yml" template.
|
||||
"""
|
||||
|
||||
def run_task(self, service: str, command: str) -> int:
|
||||
job_name = f"{service}-job"
|
||||
job = self.load_job(job_name)
|
||||
# Create a unique job name to make it deduplicate jobs and make it easier to
|
||||
|
|
@ -148,10 +138,41 @@ class K8sJobRunner(jobs.BaseJobRunner):
|
|||
sleep(5)
|
||||
return 0
|
||||
|
||||
def load_job(self, name: str) -> Any:
|
||||
"""
|
||||
Find a given job definition in the rendered k8s/jobs.yml template.
|
||||
"""
|
||||
all_jobs = self.render("k8s", "jobs.yml")
|
||||
for job in serialize.load_all(all_jobs):
|
||||
job_name = job["metadata"]["name"]
|
||||
if not isinstance(job_name, str):
|
||||
raise exceptions.TutorError(
|
||||
f"Invalid job name: '{job_name}'. Expected str."
|
||||
)
|
||||
if job_name == name:
|
||||
return job
|
||||
raise exceptions.TutorError(f"Could not find job '{name}'")
|
||||
|
||||
class K8sContext(BaseJobContext):
|
||||
def job_runner(self, config: Config) -> K8sJobRunner:
|
||||
return K8sJobRunner(self.root, config)
|
||||
def active_job_names(self) -> List[str]:
|
||||
"""
|
||||
Return a list of active job names
|
||||
Docs:
|
||||
https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.18/#list-job-v1-batch
|
||||
|
||||
This is necessary to make sure that we don't run the same job multiple times at
|
||||
the same time.
|
||||
"""
|
||||
api = K8sClients.instance().batch_api
|
||||
return [
|
||||
job.metadata.name
|
||||
for job in api.list_namespaced_job(k8s_namespace(self.config)).items
|
||||
if job.status.active
|
||||
]
|
||||
|
||||
|
||||
class K8sContext(BaseTaskContext):
|
||||
def job_runner(self, config: Config) -> K8sTaskRunner:
|
||||
return K8sTaskRunner(self.root, config)
|
||||
|
||||
|
||||
@click.group(help="Run Open edX on Kubernetes")
|
||||
|
|
@ -163,7 +184,7 @@ def k8s(context: click.Context) -> None:
|
|||
@click.command(help="Configure and run Open edX from scratch")
|
||||
@click.option("-I", "--non-interactive", is_flag=True, help="Run non-interactively")
|
||||
@click.pass_context
|
||||
def quickstart(context: click.Context, non_interactive: bool) -> None:
|
||||
def launch(context: click.Context, non_interactive: bool) -> None:
|
||||
run_upgrade_from_release = tutor_env.should_upgrade_from_release(context.obj.root)
|
||||
if run_upgrade_from_release is not None:
|
||||
click.echo(fmt.title("Upgrading from an older release"))
|
||||
|
|
@ -214,6 +235,19 @@ Press enter when you are ready to continue"""
|
|||
)
|
||||
|
||||
|
||||
@click.command(help="Configure and run Open edX from scratch")
|
||||
@click.option("-I", "--non-interactive", is_flag=True, help="Run non-interactively")
|
||||
@click.pass_context
|
||||
def quickstart(context: click.Context, non_interactive: bool) -> None:
|
||||
"""
|
||||
This command has been renamed to 'launch'.
|
||||
"""
|
||||
fmt.echo_alert(
|
||||
"The 'quickstart' command is deprecated and will be removed in a later release. Use 'launch' instead."
|
||||
)
|
||||
context.invoke(launch, non_interactive=non_interactive)
|
||||
|
||||
|
||||
@click.command(
|
||||
short_help="Run all configured Open edX resources",
|
||||
help=(
|
||||
|
|
@ -326,17 +360,34 @@ def delete(context: K8sContext, yes: bool) -> None:
|
|||
)
|
||||
|
||||
|
||||
@jobs.do_group
|
||||
@click.pass_obj
|
||||
def do(context: K8sContext) -> None:
|
||||
"""
|
||||
Run a custom job in the right container(s).
|
||||
|
||||
We make sure that some essential containers (databases, proxy) are up before we
|
||||
launch the jobs.
|
||||
"""
|
||||
|
||||
@hooks.Actions.DO_JOB.add()
|
||||
def _start_base_deployments(_job_name: str, *_args: Any, **_kwargs: Any) -> None:
|
||||
"""
|
||||
We add this logic to an action callback because we do not want to trigger it
|
||||
whenever we run `tutor k8s do <job> --help`.
|
||||
"""
|
||||
config = tutor_config.load(context.root)
|
||||
wait_for_deployment_ready(config, "caddy")
|
||||
for name in ["elasticsearch", "mysql", "mongodb"]:
|
||||
if tutor_config.is_service_activated(config, name):
|
||||
wait_for_deployment_ready(config, name)
|
||||
|
||||
|
||||
@click.command(help="Initialise all applications")
|
||||
@click.option("-l", "--limit", help="Limit initialisation to this service or plugin")
|
||||
@click.pass_obj
|
||||
def init(context: K8sContext, limit: Optional[str]) -> None:
|
||||
config = tutor_config.load(context.root)
|
||||
runner = context.job_runner(config)
|
||||
wait_for_deployment_ready(config, "caddy")
|
||||
for name in ["elasticsearch", "mysql", "mongodb"]:
|
||||
if tutor_config.is_service_activated(config, name):
|
||||
wait_for_deployment_ready(config, name)
|
||||
jobs.initialise(runner, limit_to=limit)
|
||||
@click.pass_context
|
||||
def init(context: click.Context, limit: Optional[str]) -> None:
|
||||
context.invoke(do.commands["init"], limit=limit)
|
||||
|
||||
|
||||
@click.command(help="Scale the number of replicas of a given deployment")
|
||||
|
|
@ -357,64 +408,6 @@ def scale(context: K8sContext, deployment: str, replicas: int) -> None:
|
|||
)
|
||||
|
||||
|
||||
@click.command(help="Create an Open edX user and interactively set their password")
|
||||
@click.option("--superuser", is_flag=True, help="Make superuser")
|
||||
@click.option("--staff", is_flag=True, help="Make staff user")
|
||||
@click.option(
|
||||
"-p",
|
||||
"--password",
|
||||
help="Specify password from the command line. If undefined, you will be prompted to input a password",
|
||||
prompt=True,
|
||||
hide_input=True,
|
||||
)
|
||||
@click.argument("name")
|
||||
@click.argument("email")
|
||||
@click.pass_obj
|
||||
def createuser(
|
||||
context: K8sContext,
|
||||
superuser: str,
|
||||
staff: bool,
|
||||
password: str,
|
||||
name: str,
|
||||
email: str,
|
||||
) -> None:
|
||||
config = tutor_config.load(context.root)
|
||||
command = jobs.create_user_command(superuser, staff, name, email, password=password)
|
||||
runner = context.job_runner(config)
|
||||
runner.run_job("lms", command)
|
||||
|
||||
|
||||
@click.command(help="Import the demo course")
|
||||
@click.pass_obj
|
||||
def importdemocourse(context: K8sContext) -> None:
|
||||
fmt.echo_info("Importing demo course")
|
||||
config = tutor_config.load(context.root)
|
||||
runner = context.job_runner(config)
|
||||
jobs.import_demo_course(runner)
|
||||
|
||||
|
||||
@click.command(
|
||||
help="Assign a theme to the LMS and the CMS. To reset to the default theme , use 'default' as the theme name."
|
||||
)
|
||||
@click.option(
|
||||
"-d",
|
||||
"--domain",
|
||||
"domains",
|
||||
multiple=True,
|
||||
help=(
|
||||
"Limit the theme to these domain names. By default, the theme is "
|
||||
"applied to the LMS and the CMS, both in development and production mode"
|
||||
),
|
||||
)
|
||||
@click.argument("theme_name")
|
||||
@click.pass_obj
|
||||
def settheme(context: K8sContext, domains: List[str], theme_name: str) -> None:
|
||||
config = tutor_config.load(context.root)
|
||||
runner = context.job_runner(config)
|
||||
domains = domains or jobs.get_all_openedx_domains(config)
|
||||
jobs.set_theme(theme_name, domains, runner)
|
||||
|
||||
|
||||
@click.command(
|
||||
name="exec",
|
||||
help="Execute a command in a pod of the given application",
|
||||
|
|
@ -463,7 +456,7 @@ def wait(context: K8sContext, name: str) -> None:
|
|||
|
||||
@click.command(
|
||||
short_help="Perform release-specific upgrade tasks",
|
||||
help="Perform release-specific upgrade tasks. To perform a full upgrade remember to run `quickstart`.",
|
||||
help="Perform release-specific upgrade tasks. To perform a full upgrade remember to run `launch`.",
|
||||
)
|
||||
@click.option(
|
||||
"--from",
|
||||
|
|
@ -479,7 +472,7 @@ def upgrade(context: click.Context, from_release: Optional[str]) -> None:
|
|||
else:
|
||||
fmt.echo_alert(
|
||||
"This command only performs a partial upgrade of your Open edX platform. "
|
||||
"To perform a full upgrade, you should run `tutor k8s quickstart`."
|
||||
"To perform a full upgrade, you should run `tutor k8s launch`."
|
||||
)
|
||||
upgrade_from(context.obj, from_release)
|
||||
# We update the environment to update the version
|
||||
|
|
@ -569,6 +562,7 @@ def k8s_namespace(config: Config) -> str:
|
|||
return get_typed(config, "K8S_NAMESPACE", str)
|
||||
|
||||
|
||||
k8s.add_command(launch)
|
||||
k8s.add_command(quickstart)
|
||||
k8s.add_command(start)
|
||||
k8s.add_command(stop)
|
||||
|
|
@ -576,12 +570,15 @@ k8s.add_command(reboot)
|
|||
k8s.add_command(delete)
|
||||
k8s.add_command(init)
|
||||
k8s.add_command(scale)
|
||||
k8s.add_command(createuser)
|
||||
k8s.add_command(importdemocourse)
|
||||
k8s.add_command(settheme)
|
||||
k8s.add_command(exec_command)
|
||||
k8s.add_command(logs)
|
||||
k8s.add_command(wait)
|
||||
k8s.add_command(upgrade)
|
||||
k8s.add_command(apply_command)
|
||||
k8s.add_command(status)
|
||||
|
||||
|
||||
@hooks.Actions.PLUGINS_LOADED.add()
|
||||
def _add_k8s_do_commands() -> None:
|
||||
jobs.add_job_commands(do)
|
||||
k8s.add_command(do)
|
||||
|
|
|
|||
|
|
@ -13,7 +13,7 @@ from tutor.commands.upgrade.local import upgrade_from
|
|||
from tutor.types import Config, get_typed
|
||||
|
||||
|
||||
class LocalJobRunner(compose.ComposeJobRunner):
|
||||
class LocalTaskRunner(compose.ComposeTaskRunner):
|
||||
def __init__(self, root: str, config: Config):
|
||||
"""
|
||||
Load docker-compose files from local/.
|
||||
|
|
@ -52,8 +52,8 @@ class LocalContext(compose.BaseComposeContext):
|
|||
COMPOSE_TMP_FILTER = hooks.Filters.COMPOSE_LOCAL_TMP
|
||||
COMPOSE_JOBS_TMP_FILTER = hooks.Filters.COMPOSE_LOCAL_JOBS_TMP
|
||||
|
||||
def job_runner(self, config: Config) -> LocalJobRunner:
|
||||
return LocalJobRunner(self.root, config)
|
||||
def job_runner(self, config: Config) -> LocalTaskRunner:
|
||||
return LocalTaskRunner(self.root, config)
|
||||
|
||||
|
||||
@click.group(help="Run Open edX locally with docker-compose")
|
||||
|
|
@ -67,7 +67,7 @@ def local(context: click.Context) -> None:
|
|||
@click.option("-I", "--non-interactive", is_flag=True, help="Run non-interactively")
|
||||
@click.option("-p", "--pullimages", is_flag=True, help="Update docker images")
|
||||
@click.pass_context
|
||||
def quickstart(
|
||||
def launch(
|
||||
context: click.Context,
|
||||
mounts: t.Tuple[t.List[compose.MountParam.MountType]],
|
||||
non_interactive: bool,
|
||||
|
|
@ -142,7 +142,7 @@ Press enter when you are ready to continue"""
|
|||
click.echo(fmt.title("Starting the platform in detached mode"))
|
||||
context.invoke(compose.start, detach=True)
|
||||
click.echo(fmt.title("Database creation and migrations"))
|
||||
context.invoke(compose.init)
|
||||
context.invoke(compose.do.commands["init"])
|
||||
|
||||
config = tutor_config.load(context.obj.root)
|
||||
fmt.echo_info(
|
||||
|
|
@ -159,9 +159,31 @@ Your Open edX platform is ready and can be accessed at the following urls:
|
|||
)
|
||||
|
||||
|
||||
@click.command(help="Deprecated alias to 'launch'")
|
||||
@compose.mount_option
|
||||
@click.option("-I", "--non-interactive", is_flag=True, help="Run non-interactively")
|
||||
@click.option("-p", "--pullimages", is_flag=True, help="Update docker images")
|
||||
@click.pass_context
|
||||
def quickstart(
|
||||
context: click.Context,
|
||||
mounts: t.Tuple[t.List[compose.MountParam.MountType]],
|
||||
non_interactive: bool,
|
||||
pullimages: bool,
|
||||
) -> None:
|
||||
"""
|
||||
This command has been renamed to 'launch'.
|
||||
"""
|
||||
fmt.echo_alert(
|
||||
"The 'quickstart' command is deprecated and will be removed in a later release. Use 'launch' instead."
|
||||
)
|
||||
context.invoke(
|
||||
launch, non_interactive=non_interactive, pullimages=pullimages, mounts=mounts
|
||||
)
|
||||
|
||||
|
||||
@click.command(
|
||||
short_help="Perform release-specific upgrade tasks",
|
||||
help="Perform release-specific upgrade tasks. To perform a full upgrade remember to run `quickstart`.",
|
||||
help="Perform release-specific upgrade tasks. To perform a full upgrade remember to run `launch`.",
|
||||
)
|
||||
@click.option(
|
||||
"--from",
|
||||
|
|
@ -172,7 +194,7 @@ Your Open edX platform is ready and can be accessed at the following urls:
|
|||
def upgrade(context: click.Context, from_release: t.Optional[str]) -> None:
|
||||
fmt.echo_alert(
|
||||
"This command only performs a partial upgrade of your Open edX platform. "
|
||||
"To perform a full upgrade, you should run `tutor local quickstart`."
|
||||
"To perform a full upgrade, you should run `tutor local launch`."
|
||||
)
|
||||
if from_release is None:
|
||||
from_release = tutor_env.get_env_release(context.obj.root)
|
||||
|
|
@ -190,11 +212,12 @@ def _stop_on_dev_start(root: str, config: Config, project_name: str) -> None:
|
|||
Stop the local platform as soon as a platform with a different project name is
|
||||
started.
|
||||
"""
|
||||
runner = LocalJobRunner(root, config)
|
||||
runner = LocalTaskRunner(root, config)
|
||||
if project_name != runner.project_name:
|
||||
runner.docker_compose("stop")
|
||||
|
||||
|
||||
local.add_command(launch)
|
||||
local.add_command(quickstart)
|
||||
local.add_command(upgrade)
|
||||
compose.add_commands(local)
|
||||
|
|
|
|||
|
|
@ -19,7 +19,7 @@ def load(root: str) -> Config:
|
|||
raise exceptions.TutorError(
|
||||
"Project root does not exist. Make sure to generate the initial "
|
||||
"configuration with `tutor config save --interactive` or `tutor local "
|
||||
"quickstart` prior to running other commands."
|
||||
"launch` prior to running other commands."
|
||||
)
|
||||
env.check_is_up_to_date(root)
|
||||
return load_full(root)
|
||||
|
|
@ -139,7 +139,7 @@ def get_template(filename: str) -> Config:
|
|||
|
||||
Entries in this configuration are unrendered.
|
||||
"""
|
||||
config = serialize.load(env.read_template_file("config", filename))
|
||||
config = serialize.load(env.read_core_template_file("config", filename))
|
||||
return cast_config(config)
|
||||
|
||||
|
||||
|
|
@ -220,10 +220,9 @@ def upgrade_obsolete(config: Config) -> None:
|
|||
]:
|
||||
if name in config:
|
||||
config[name.replace("ACTIVATE_", "RUN_")] = config.pop(name)
|
||||
# Replace RUN_CADDY by ENABLE_WEB_PROXY
|
||||
# Replace nginx by caddy
|
||||
if "RUN_CADDY" in config:
|
||||
config["ENABLE_WEB_PROXY"] = config.pop("RUN_CADDY")
|
||||
# Replace RUN_CADDY by ENABLE_WEB_PROXY
|
||||
if "NGINX_HTTP_PORT" in config:
|
||||
config["CADDY_HTTP_PORT"] = config.pop("NGINX_HTTP_PORT")
|
||||
|
||||
|
|
|
|||
56
tutor/env.py
56
tutor/env.py
|
|
@ -60,21 +60,35 @@ _prepare_environment()
|
|||
|
||||
|
||||
class JinjaEnvironment(jinja2.Environment):
|
||||
loader: jinja2.BaseLoader
|
||||
loader: jinja2.FileSystemLoader
|
||||
|
||||
def __init__(self, template_roots: t.List[str]) -> None:
|
||||
def __init__(self) -> None:
|
||||
template_roots = hooks.Filters.ENV_TEMPLATE_ROOTS.apply([TEMPLATES_ROOT])
|
||||
loader = jinja2.FileSystemLoader(template_roots)
|
||||
super().__init__(loader=loader, undefined=jinja2.StrictUndefined)
|
||||
|
||||
def read_str(self, template_name: str) -> str:
|
||||
return self.read_bytes(template_name).decode()
|
||||
|
||||
def read_bytes(self, template_name: str) -> bytes:
|
||||
with open(self.find_os_path(template_name), "rb") as f:
|
||||
return f.read()
|
||||
|
||||
def find_os_path(self, template_name: str) -> str:
|
||||
path = template_name.replace("/", os.sep)
|
||||
for templates_root in self.loader.searchpath:
|
||||
full_path = os.path.join(templates_root, path)
|
||||
if os.path.exists(full_path):
|
||||
return full_path
|
||||
raise ValueError("Template path does not exist")
|
||||
|
||||
|
||||
class Renderer:
|
||||
def __init__(self, config: t.Optional[Config] = None):
|
||||
config = config or {}
|
||||
self.config = deepcopy(config)
|
||||
self.template_roots = hooks.Filters.ENV_TEMPLATE_ROOTS.apply([TEMPLATES_ROOT])
|
||||
self.config = deepcopy(config or {})
|
||||
|
||||
# Create environment with extra filters and globals
|
||||
self.environment = JinjaEnvironment(self.template_roots)
|
||||
self.environment = JinjaEnvironment()
|
||||
|
||||
# Filters
|
||||
plugin_filters = hooks.Filters.ENV_TEMPLATE_FILTERS.iterate()
|
||||
|
|
@ -136,14 +150,6 @@ class Renderer:
|
|||
"""
|
||||
yield from self.iter_templates_in(subdir)
|
||||
|
||||
def find_os_path(self, template_name: str) -> str:
|
||||
path = template_name.replace("/", os.sep)
|
||||
for templates_root in self.template_roots:
|
||||
full_path = os.path.join(templates_root, path)
|
||||
if os.path.exists(full_path):
|
||||
return full_path
|
||||
raise ValueError("Template path does not exist")
|
||||
|
||||
def patch(self, name: str, separator: str = "\n", suffix: str = "") -> str:
|
||||
"""
|
||||
Render calls to {{ patch("...") }} in environment templates from plugin patches.
|
||||
|
|
@ -174,8 +180,7 @@ class Renderer:
|
|||
"""
|
||||
if is_binary_file(template_name):
|
||||
# Don't try to render binary files
|
||||
with open(self.find_os_path(template_name), "rb") as f:
|
||||
return f.read()
|
||||
return self.environment.read_bytes(template_name)
|
||||
|
||||
try:
|
||||
template = self.environment.get_template(template_name)
|
||||
|
|
@ -397,9 +402,17 @@ def current_version(root: str) -> t.Optional[str]:
|
|||
def read_template_file(*path: str) -> str:
|
||||
"""
|
||||
Read raw content of template located at `path`.
|
||||
|
||||
The template may be located inside any of the template root folders.
|
||||
"""
|
||||
src = template_path(*path)
|
||||
with open(src, encoding="utf-8") as fi:
|
||||
return JinjaEnvironment().read_str("/".join(path))
|
||||
|
||||
|
||||
def read_core_template_file(*path: str) -> str:
|
||||
"""
|
||||
Read raw content of template located in tutor core template directory.
|
||||
"""
|
||||
with open(os.path.join(TEMPLATES_ROOT, *path), encoding="utf-8") as fi:
|
||||
return fi.read()
|
||||
|
||||
|
||||
|
|
@ -408,13 +421,6 @@ def is_binary_file(path: str) -> bool:
|
|||
return ext in BIN_FILE_EXTENSIONS
|
||||
|
||||
|
||||
def template_path(*path: str, templates_root: str = TEMPLATES_ROOT) -> str:
|
||||
"""
|
||||
Return the template file's absolute path.
|
||||
"""
|
||||
return os.path.join(templates_root, *path)
|
||||
|
||||
|
||||
def data_path(root: str, *path: str) -> str:
|
||||
"""
|
||||
Return the file's absolute path inside the data directory.
|
||||
|
|
|
|||
|
|
@ -4,7 +4,7 @@ __license__ = "Apache 2.0"
|
|||
import typing as t
|
||||
|
||||
# These imports are the hooks API
|
||||
from . import actions, contexts, filters
|
||||
from . import actions, contexts, filters, priorities
|
||||
from .consts import *
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -6,6 +6,7 @@ import typing as t
|
|||
|
||||
from typing_extensions import ParamSpec
|
||||
|
||||
from . import priorities
|
||||
from .contexts import Contextualized
|
||||
|
||||
P = ParamSpec("P")
|
||||
|
|
@ -13,8 +14,6 @@ P = ParamSpec("P")
|
|||
# the future.
|
||||
# CallableAction = t.Callable[P, None]
|
||||
|
||||
DEFAULT_PRIORITY = 10
|
||||
|
||||
|
||||
class ActionCallback(Contextualized, t.Generic[P]):
|
||||
def __init__(
|
||||
|
|
@ -24,7 +23,7 @@ class ActionCallback(Contextualized, t.Generic[P]):
|
|||
):
|
||||
super().__init__()
|
||||
self.func = func
|
||||
self.priority = priority or DEFAULT_PRIORITY
|
||||
self.priority = priority or priorities.DEFAULT
|
||||
|
||||
def do(
|
||||
self,
|
||||
|
|
@ -81,15 +80,7 @@ class Action(t.Generic[P]):
|
|||
|
||||
def inner(func: t.Callable[P, None]) -> t.Callable[P, None]:
|
||||
callback = ActionCallback(func, priority=priority)
|
||||
# I wish we could use bisect.insort_right here but the `key=` parameter
|
||||
# is unsupported in Python 3.9
|
||||
position = 0
|
||||
while (
|
||||
position < len(self.callbacks)
|
||||
and self.callbacks[position].priority <= callback.priority
|
||||
):
|
||||
position += 1
|
||||
self.callbacks.insert(position, callback)
|
||||
priorities.insert_callback(callback, self.callbacks)
|
||||
return func
|
||||
|
||||
return inner
|
||||
|
|
@ -182,8 +173,7 @@ def get_template(name: str) -> ActionTemplate[t.Any]:
|
|||
|
||||
|
||||
def add(
|
||||
name: str,
|
||||
priority: t.Optional[int] = None,
|
||||
name: str, priority: t.Optional[int] = None
|
||||
) -> t.Callable[[t.Callable[P, None]], t.Callable[P, None]]:
|
||||
"""
|
||||
Decorator to add a callback action associated to a name.
|
||||
|
|
@ -193,8 +183,8 @@ def add(
|
|||
:py:class:`tutor.hooks.Actions` instead.
|
||||
:param priority: optional order in which the action callbacks are performed. Higher
|
||||
values mean that they will be performed later. The default value is
|
||||
``DEFAULT_PRIORITY`` (10). Actions that should be performed last should
|
||||
have a priority of 100.
|
||||
``priorities.DEFAULT`` (10). Actions that should be performed last should have a
|
||||
priority of 100.
|
||||
|
||||
Usage::
|
||||
|
||||
|
|
|
|||
|
|
@ -7,7 +7,7 @@ from __future__ import annotations
|
|||
# The Tutor plugin system is licensed under the terms of the Apache 2.0 license.
|
||||
__license__ = "Apache 2.0"
|
||||
|
||||
from typing import Any, Callable
|
||||
from typing import Any, Callable, Iterable
|
||||
|
||||
import click
|
||||
|
||||
|
|
@ -59,6 +59,13 @@ class Actions:
|
|||
#: This action does not have any parameter.
|
||||
CORE_READY: Action[[]] = actions.get("core:ready")
|
||||
|
||||
#: Called just before triggering the job tasks of any `... do <job>` command.
|
||||
#:
|
||||
#: :parameter: str job: job name.
|
||||
#: :parameter: args: job positional arguments.
|
||||
#: :parameter: kwargs: job named arguments.
|
||||
DO_JOB: Action[[str, Any]] = actions.get("do:job")
|
||||
|
||||
#: Called as soon as we have access to the Tutor project root.
|
||||
#:
|
||||
#: :parameter str root: absolute path to the project root.
|
||||
|
|
@ -116,6 +123,35 @@ class Filters:
|
|||
return items
|
||||
"""
|
||||
|
||||
#: List of command line interface (CLI) commands.
|
||||
#:
|
||||
#: :parameter list commands: commands are instances of ``click.Command``. They will
|
||||
#: all be added as subcommands of the main ``tutor`` command.
|
||||
CLI_COMMANDS: Filter[list[click.Command], []] = filters.get("cli:commands")
|
||||
|
||||
#: List of `do ...` commands.
|
||||
#:
|
||||
#: :parameter list commands: see :py:data:`CLI_COMMANDS`. These commands will be
|
||||
#: added as subcommands to the `local/dev/k8s do` commands. They must return a list of
|
||||
#: ("service name", "service command") tuples. Each "service command" will be executed
|
||||
#: in the "service" container, both in local, dev and k8s mode.
|
||||
CLI_DO_COMMANDS: Filter[
|
||||
list[Callable[[Any], Iterable[tuple[str, str]]]], []
|
||||
] = filters.get("cli:commands:do")
|
||||
|
||||
#: List of initialization tasks (scripts) to be run in the `init` job. This job
|
||||
#: includes all database migrations, setting up, etc. To run some tasks before or
|
||||
#: after others, they should be assigned a different priority.
|
||||
#:
|
||||
#: :parameter list[tuple[str, str]] tasks: list of ``(service, task)`` tuples. Each
|
||||
#: task is essentially a bash script to be run in the "service" container. Scripts
|
||||
#: may contain Jinja markup, similar to templates.
|
||||
CLI_DO_INIT_TASKS: Filter[list[tuple[str, str]], []] = filters.get(
|
||||
"cli:commands:do:init"
|
||||
)
|
||||
|
||||
#: DEPRECATED use :py:data:`CLI_DO_INIT_TASKS` instead.
|
||||
#:
|
||||
#: List of commands to be executed during initialization. These commands typically
|
||||
#: include database migrations, setting feature flags, etc.
|
||||
#:
|
||||
|
|
@ -123,16 +159,19 @@ class Filters:
|
|||
#:
|
||||
#: - ``service`` is the name of the container in which the task will be executed.
|
||||
#: - ``path`` is a tuple that corresponds to a template relative path.
|
||||
#: Example: ``("myplugin", "hooks", "myservice", "pre-init")`` (see:py:data:`IMAGES_BUILD`).
|
||||
#: Example: ``("myplugin", "hooks", "myservice", "pre-init")`` (see :py:data:`IMAGES_BUILD`).
|
||||
#: The command to execute will be read from that template, after it is rendered.
|
||||
COMMANDS_INIT: Filter[list[tuple[str, tuple[str, ...]]], str] = filters.get(
|
||||
"commands:init"
|
||||
)
|
||||
|
||||
#: DEPRECATED use :py:data:`CLI_DO_INIT_TASKS` instead with a lower priority score.
|
||||
#:
|
||||
#: List of commands to be executed prior to initialization. These commands are run even
|
||||
#: before the mysql databases are created and the migrations are applied.
|
||||
#:
|
||||
#: :parameter list[tuple[str, tuple[str, ...]]] tasks: list of ``(service, path)`` tasks. (see :py:data:`COMMANDS_INIT`).
|
||||
#: :parameter list[tuple[str, tuple[str, ...]]] tasks: list of ``(service, path)``
|
||||
#: tasks. (see :py:data:`COMMANDS_INIT`).
|
||||
COMMANDS_PRE_INIT: Filter[list[tuple[str, tuple[str, ...]]], []] = filters.get(
|
||||
"commands:pre-init"
|
||||
)
|
||||
|
|
@ -205,12 +244,6 @@ class Filters:
|
|||
#: Parameters are the same as for :py:data:`IMAGES_PULL`.
|
||||
IMAGES_PUSH: Filter[list[tuple[str, str]], [Config]] = filters.get("images:push")
|
||||
|
||||
#: List of command line interface (CLI) commands.
|
||||
#:
|
||||
#: :parameter list commands: commands are instances of ``click.Command``. They will
|
||||
#: all be added as subcommands of the main ``tutor`` command.
|
||||
CLI_COMMANDS: Filter[list[click.Command], []] = filters.get("cli:commands")
|
||||
|
||||
#: Declare new default configuration settings that don't necessarily have to be saved in the user
|
||||
#: ``config.yml`` file. Default settings may be overridden with ``tutor config save --set=...``, in which
|
||||
#: case they will automatically be added to ``config.yml``.
|
||||
|
|
@ -324,6 +357,34 @@ class Filters:
|
|||
"env:templates:variables"
|
||||
)
|
||||
|
||||
#: List of images to be built when we run ``tutor images build ...``.
|
||||
#:
|
||||
#: :parameter list[tuple[str, tuple[str, ...], str, tuple[str, ...]]] tasks: list of ``(name, path, tag, args)`` tuples.
|
||||
#:
|
||||
#: - ``name`` is the name of the image, as in ``tutor images build myimage``.
|
||||
#: - ``path`` is the relative path to the folder that contains the Dockerfile.
|
||||
#: For instance ``("myplugin", "build", "myservice")`` indicates that the template will be read from
|
||||
#: ``myplugin/build/myservice/Dockerfile``
|
||||
#: - ``tag`` is the Docker tag that will be applied to the image. It will be
|
||||
#: rendered at runtime with the user configuration. Thus, the image tag could
|
||||
#: be ``"{{ DOCKER_REGISTRY }}/myimage:{{ TUTOR_VERSION }}"``.
|
||||
#: - ``args`` is a list of arguments that will be passed to ``docker build ...``.
|
||||
#: :parameter dict config: user configuration.
|
||||
IMAGES_BUILD = filters.get("images:build")
|
||||
|
||||
#: List of images to be pulled when we run ``tutor images pull ...``.
|
||||
#:
|
||||
#: :parameter list[tuple[str, str]] tasks: list of ``(name, tag)`` tuples.
|
||||
#:
|
||||
#: - ``name`` is the name of the image, as in ``tutor images pull myimage``.
|
||||
#: - ``tag`` is the Docker tag that will be applied to the image. (see :py:data:`IMAGES_BUILD`).
|
||||
#: :parameter dict config: user configuration.
|
||||
IMAGES_PULL = filters.get("images:pull")
|
||||
|
||||
#: List of images to be pushed when we run ``tutor images push ...``.
|
||||
#: Parameters are the same as for :py:data:`IMAGES_PULL`.
|
||||
IMAGES_PUSH = filters.get("images:push")
|
||||
|
||||
#: List of installed plugins. In order to be added to this list, a plugin must first
|
||||
#: be discovered (see :py:data:`Actions.CORE_READY`).
|
||||
#:
|
||||
|
|
|
|||
|
|
@ -6,7 +6,7 @@ import typing as t
|
|||
|
||||
from typing_extensions import Concatenate, ParamSpec
|
||||
|
||||
from . import contexts
|
||||
from . import contexts, priorities
|
||||
|
||||
T = t.TypeVar("T")
|
||||
P = ParamSpec("P")
|
||||
|
|
@ -19,9 +19,12 @@ E = t.TypeVar("E")
|
|||
|
||||
|
||||
class FilterCallback(contexts.Contextualized, t.Generic[T, P]):
|
||||
def __init__(self, func: t.Callable[Concatenate[T, P], T]):
|
||||
def __init__(
|
||||
self, func: t.Callable[Concatenate[T, P], T], priority: t.Optional[int] = None
|
||||
):
|
||||
super().__init__()
|
||||
self.func = func
|
||||
self.priority = priority or priorities.DEFAULT
|
||||
|
||||
def apply(self, value: T, *args: P.args, **kwargs: P.kwargs) -> T:
|
||||
return self.func(value, *args, **kwargs)
|
||||
|
|
@ -72,14 +75,15 @@ class Filter(t.Generic[T, P]):
|
|||
return cls.INDEX.setdefault(name, cls(name))
|
||||
|
||||
def add(
|
||||
self,
|
||||
self, priority: t.Optional[int] = None
|
||||
) -> t.Callable[
|
||||
[t.Callable[Concatenate[T, P], T]], t.Callable[Concatenate[T, P], T]
|
||||
]:
|
||||
def inner(
|
||||
func: t.Callable[Concatenate[T, P], T]
|
||||
) -> t.Callable[Concatenate[T, P], T]:
|
||||
self.callbacks.append(FilterCallback[T, P](func))
|
||||
callback = FilterCallback(func, priority=priority)
|
||||
priorities.insert_callback(callback, self.callbacks)
|
||||
return func
|
||||
|
||||
return inner
|
||||
|
|
@ -138,10 +142,14 @@ class Filter(t.Generic[T, P]):
|
|||
]
|
||||
|
||||
# The methods below are specific to filters which take lists as first arguments
|
||||
def add_item(self: "Filter[t.List[E], P]", item: E) -> None:
|
||||
self.add_items([item])
|
||||
def add_item(
|
||||
self: "Filter[t.List[E], P]", item: E, priority: t.Optional[int] = None
|
||||
) -> None:
|
||||
self.add_items([item], priority=priority)
|
||||
|
||||
def add_items(self: "Filter[t.List[E], P]", items: t.List[E]) -> None:
|
||||
def add_items(
|
||||
self: "Filter[t.List[E], P]", items: t.List[E], priority: t.Optional[int] = None
|
||||
) -> None:
|
||||
# Unfortunately we have to type-ignore this line. If not, mypy complains with:
|
||||
#
|
||||
# Argument 1 has incompatible type "Callable[[Arg(List[E], 'values'), **P], List[E]]"; expected "Callable[[List[E], **P], List[E]]"
|
||||
|
|
@ -149,7 +157,7 @@ class Filter(t.Generic[T, P]):
|
|||
#
|
||||
# But we are unable to mark arguments positional-only (by adding / after values arg) in Python 3.7.
|
||||
# Get rid of this statement after Python 3.7 EOL.
|
||||
@self.add() # type: ignore
|
||||
@self.add(priority=priority) # type: ignore
|
||||
def callback(
|
||||
values: t.List[E], *_args: P.args, **_kwargs: P.kwargs
|
||||
) -> t.List[E]:
|
||||
|
|
@ -211,12 +219,16 @@ def get_template(name: str) -> FilterTemplate[t.Any, t.Any]:
|
|||
|
||||
|
||||
def add(
|
||||
name: str,
|
||||
name: str, priority: t.Optional[int] = None
|
||||
) -> t.Callable[[t.Callable[Concatenate[T, P], T]], t.Callable[Concatenate[T, P], T]]:
|
||||
"""
|
||||
Decorator for functions that will be applied to a single named filter.
|
||||
|
||||
:param name: name of the filter to which the decorated function should be added.
|
||||
:param str name: name of the filter to which the decorated function should be added.
|
||||
:param int priority: optional order in which the filter callbacks are called. Higher
|
||||
values mean that they will be performed later. The default value is
|
||||
``priorities.DEFAULT`` (10). Filters that should be called last should have a
|
||||
priority of 100.
|
||||
|
||||
The return value of each filter function callback will be passed as the first argument to the next one.
|
||||
|
||||
|
|
@ -233,15 +245,16 @@ def add(
|
|||
# After filters have been created, the result of calling all filter callbacks is obtained by running:
|
||||
hooks.filters.apply("my-filter", initial_value, some_other_argument_value)
|
||||
"""
|
||||
return Filter.get(name).add()
|
||||
return Filter.get(name).add(priority=priority)
|
||||
|
||||
|
||||
def add_item(name: str, item: T) -> None:
|
||||
def add_item(name: str, item: T, priority: t.Optional[int] = None) -> None:
|
||||
"""
|
||||
Convenience function to add a single item to a filter that returns a list of items.
|
||||
|
||||
:param name: filter name.
|
||||
:param object item: item that will be appended to the resulting list.
|
||||
:param int priority: see :py:data:`add`.
|
||||
|
||||
Usage::
|
||||
|
||||
|
|
@ -252,10 +265,10 @@ def add_item(name: str, item: T) -> None:
|
|||
|
||||
assert ["item1", "item2"] == hooks.filters.apply("my-filter", [])
|
||||
"""
|
||||
get(name).add_item(item)
|
||||
get(name).add_item(item, priority=priority)
|
||||
|
||||
|
||||
def add_items(name: str, items: t.List[T]) -> None:
|
||||
def add_items(name: str, items: t.List[T], priority: t.Optional[int] = None) -> None:
|
||||
"""
|
||||
Convenience function to add multiple item to a filter that returns a list of items.
|
||||
|
||||
|
|
@ -270,7 +283,7 @@ def add_items(name: str, items: t.List[T]) -> None:
|
|||
|
||||
assert ["item1", "item2"] == hooks.filters.apply("my-filter", [])
|
||||
"""
|
||||
get(name).add_items(items)
|
||||
get(name).add_items(items, priority=priority)
|
||||
|
||||
|
||||
def iterate(name: str, *args: t.Any, **kwargs: t.Any) -> t.Iterator[T]:
|
||||
|
|
|
|||
|
|
@ -0,0 +1,25 @@
|
|||
import typing as t
|
||||
|
||||
from typing_extensions import Protocol
|
||||
|
||||
HIGH = 5
|
||||
DEFAULT = 10
|
||||
LOW = 50
|
||||
|
||||
|
||||
class PrioritizedCallback(Protocol):
|
||||
priority: int
|
||||
|
||||
|
||||
TPrioritized = t.TypeVar("TPrioritized", bound=PrioritizedCallback)
|
||||
|
||||
|
||||
def insert_callback(callback: TPrioritized, callbacks: t.List[TPrioritized]) -> None:
|
||||
# I wish we could use bisect.insort_right here but the `key=` parameter
|
||||
# is unsupported in Python 3.9
|
||||
position = 0
|
||||
while (
|
||||
position < len(callbacks) and callbacks[position].priority <= callback.priority
|
||||
):
|
||||
position += 1
|
||||
callbacks.insert(position, callback)
|
||||
157
tutor/jobs.py
157
tutor/jobs.py
|
|
@ -1,157 +0,0 @@
|
|||
import typing as t
|
||||
|
||||
from tutor import env, fmt, hooks
|
||||
from tutor.types import Config, get_typed
|
||||
|
||||
BASE_OPENEDX_COMMAND = """
|
||||
echo "Loading settings $DJANGO_SETTINGS_MODULE"
|
||||
"""
|
||||
|
||||
|
||||
class BaseJobRunner:
|
||||
"""
|
||||
A job runner is responsible for getting a certain task to complete.
|
||||
"""
|
||||
|
||||
def __init__(self, root: str, config: Config):
|
||||
self.root = root
|
||||
self.config = config
|
||||
|
||||
def run_job_from_template(self, service: str, *path: str) -> None:
|
||||
command = self.render(*path)
|
||||
self.run_job(service, command)
|
||||
|
||||
def render(self, *path: str) -> str:
|
||||
rendered = env.render_file(self.config, *path).strip()
|
||||
if isinstance(rendered, bytes):
|
||||
raise TypeError("Cannot load job from binary file")
|
||||
return rendered
|
||||
|
||||
def run_job(self, service: str, command: str) -> int:
|
||||
"""
|
||||
Given a (potentially large) string command, run it with the
|
||||
corresponding service. Implementations will differ depending on the
|
||||
deployment strategy.
|
||||
"""
|
||||
raise NotImplementedError
|
||||
|
||||
|
||||
class BaseComposeJobRunner(BaseJobRunner):
|
||||
def docker_compose(self, *command: str) -> int:
|
||||
raise NotImplementedError
|
||||
|
||||
|
||||
@hooks.Actions.CORE_READY.add()
|
||||
def _add_core_init_tasks() -> None:
|
||||
"""
|
||||
Declare core init scripts at runtime.
|
||||
|
||||
The context is important, because it allows us to select the init scripts based on
|
||||
the --limit argument.
|
||||
"""
|
||||
with hooks.Contexts.APP("mysql").enter():
|
||||
hooks.Filters.COMMANDS_INIT.add_item(("mysql", ("hooks", "mysql", "init")))
|
||||
with hooks.Contexts.APP("lms").enter():
|
||||
hooks.Filters.COMMANDS_INIT.add_item(("lms", ("hooks", "lms", "init")))
|
||||
with hooks.Contexts.APP("cms").enter():
|
||||
hooks.Filters.COMMANDS_INIT.add_item(("cms", ("hooks", "cms", "init")))
|
||||
|
||||
|
||||
def initialise(runner: BaseJobRunner, limit_to: t.Optional[str] = None) -> None:
|
||||
fmt.echo_info("Initialising all services...")
|
||||
filter_context = hooks.Contexts.APP(limit_to).name if limit_to else None
|
||||
|
||||
# Pre-init tasks
|
||||
for service, path in hooks.Filters.COMMANDS_PRE_INIT.iterate_from_context(
|
||||
filter_context
|
||||
):
|
||||
fmt.echo_info(f"Running pre-init task: {'/'.join(path)}")
|
||||
runner.run_job_from_template(service, *path)
|
||||
|
||||
# Init tasks
|
||||
for service, path in hooks.Filters.COMMANDS_INIT.iterate_from_context(
|
||||
filter_context
|
||||
):
|
||||
fmt.echo_info(f"Running init task: {'/'.join(path)}")
|
||||
runner.run_job_from_template(service, *path)
|
||||
|
||||
fmt.echo_info("All services initialised.")
|
||||
|
||||
|
||||
def create_user_command(
|
||||
superuser: str,
|
||||
staff: bool,
|
||||
username: str,
|
||||
email: str,
|
||||
password: t.Optional[str] = None,
|
||||
) -> str:
|
||||
command = BASE_OPENEDX_COMMAND
|
||||
|
||||
opts = ""
|
||||
if superuser:
|
||||
opts += " --superuser"
|
||||
if staff:
|
||||
opts += " --staff"
|
||||
command += """
|
||||
./manage.py lms manage_user {opts} {username} {email}
|
||||
"""
|
||||
if password:
|
||||
command += """
|
||||
./manage.py lms shell -c "from django.contrib.auth import get_user_model
|
||||
u = get_user_model().objects.get(username='{username}')
|
||||
u.set_password('{password}')
|
||||
u.save()"
|
||||
"""
|
||||
else:
|
||||
command += """
|
||||
./manage.py lms changepassword {username}
|
||||
"""
|
||||
|
||||
return command.format(opts=opts, username=username, email=email, password=password)
|
||||
|
||||
|
||||
def import_demo_course(runner: BaseJobRunner) -> None:
|
||||
runner.run_job_from_template("cms", "hooks", "cms", "importdemocourse")
|
||||
|
||||
|
||||
def set_theme(
|
||||
theme_name: str, domain_names: t.List[str], runner: BaseJobRunner
|
||||
) -> None:
|
||||
"""
|
||||
For each domain, get or create a Site object and assign the selected theme.
|
||||
"""
|
||||
if not domain_names:
|
||||
return
|
||||
python_code = "from django.contrib.sites.models import Site"
|
||||
for domain_name in domain_names:
|
||||
if len(domain_name) > 50:
|
||||
fmt.echo_alert(
|
||||
"Assigning a theme to a site with a long (> 50 characters) domain name."
|
||||
" The displayed site name will be truncated to 50 characters."
|
||||
)
|
||||
python_code += """
|
||||
print('Assigning theme {theme_name} to {domain_name}...')
|
||||
site, _ = Site.objects.get_or_create(domain='{domain_name}')
|
||||
if not site.name:
|
||||
name_max_length = Site._meta.get_field('name').max_length
|
||||
name = '{domain_name}'[:name_max_length]
|
||||
site.name = name
|
||||
site.save()
|
||||
site.themes.all().delete()
|
||||
site.themes.create(theme_dir_name='{theme_name}')
|
||||
""".format(
|
||||
theme_name=theme_name, domain_name=domain_name
|
||||
)
|
||||
command = BASE_OPENEDX_COMMAND + f'./manage.py lms shell -c "{python_code}"'
|
||||
runner.run_job("lms", command)
|
||||
|
||||
|
||||
def get_all_openedx_domains(config: Config) -> t.List[str]:
|
||||
return [
|
||||
get_typed(config, "LMS_HOST", str),
|
||||
get_typed(config, "LMS_HOST", str) + ":8000",
|
||||
get_typed(config, "CMS_HOST", str),
|
||||
get_typed(config, "CMS_HOST", str) + ":8001",
|
||||
get_typed(config, "PREVIEW_LMS_HOST", str),
|
||||
get_typed(config, "PREVIEW_LMS_HOST", str) + ":8000",
|
||||
]
|
||||
|
|
@ -7,7 +7,7 @@ from glob import glob
|
|||
import click
|
||||
import pkg_resources
|
||||
|
||||
from tutor import exceptions, fmt, hooks, serialize
|
||||
from tutor import env, exceptions, fmt, hooks, serialize
|
||||
from tutor.__about__ import __app__
|
||||
from tutor.types import Config
|
||||
|
||||
|
|
@ -179,12 +179,18 @@ class BasePlugin:
|
|||
)
|
||||
# Pre-init scripts: hooks = {"pre-init": ["myservice1", "myservice2"]}
|
||||
for service in pre_init_tasks:
|
||||
path = (self.name, "hooks", service, "pre-init")
|
||||
hooks.Filters.COMMANDS_PRE_INIT.add_item((service, path))
|
||||
hooks.Filters.CLI_DO_INIT_TASKS.add_item(
|
||||
(
|
||||
service,
|
||||
env.read_template_file(self.name, "hooks", service, "pre-init"),
|
||||
),
|
||||
priority=hooks.priorities.HIGH,
|
||||
)
|
||||
# Init scripts: hooks = {"init": ["myservice1", "myservice2"]}
|
||||
for service in init_tasks:
|
||||
path = (self.name, "hooks", service, "init")
|
||||
hooks.Filters.COMMANDS_INIT.add_item((service, path))
|
||||
hooks.Filters.CLI_DO_INIT_TASKS.add_item(
|
||||
(service, env.read_template_file(self.name, "hooks", service, "init"))
|
||||
)
|
||||
|
||||
def _load_templates_root(self) -> None:
|
||||
templates_root = get_callable_attr(self.obj, "templates", default=None)
|
||||
|
|
|
|||
|
|
@ -0,0 +1,42 @@
|
|||
from tutor import env
|
||||
from tutor.types import Config
|
||||
|
||||
|
||||
class BaseTaskRunner:
|
||||
"""
|
||||
A task runner is responsible for running bash commands in the right context.
|
||||
|
||||
Commands may be loaded from string or template files. The `run_task` method must be
|
||||
implemented by child classes.
|
||||
"""
|
||||
|
||||
def __init__(self, root: str, config: Config):
|
||||
self.root = root
|
||||
self.config = config
|
||||
|
||||
def run_task_from_template(self, service: str, *path: str) -> None:
|
||||
command = self.render(*path)
|
||||
self.run_task(service, command)
|
||||
|
||||
def run_task_from_str(self, service: str, command: str) -> None:
|
||||
rendered = env.render_str(self.config, command).strip()
|
||||
self.run_task(service, rendered)
|
||||
|
||||
def render(self, *path: str) -> str:
|
||||
rendered = env.render_file(self.config, *path).strip()
|
||||
if isinstance(rendered, bytes):
|
||||
raise TypeError("Cannot load job from binary file")
|
||||
return rendered
|
||||
|
||||
def run_task(self, service: str, command: str) -> int:
|
||||
"""
|
||||
Given a (potentially large) string command, run it with the
|
||||
corresponding service. Implementations will differ depending on the
|
||||
deployment strategy.
|
||||
"""
|
||||
raise NotImplementedError
|
||||
|
||||
|
||||
class BaseComposeTaskRunner(BaseTaskRunner):
|
||||
def docker_compose(self, *command: str) -> int:
|
||||
raise NotImplementedError
|
||||
|
|
@ -47,21 +47,6 @@ RUN git config --global user.email "tutor@overhang.io" \
|
|||
{{ patch("openedx-dockerfile-git-patches-default") }}
|
||||
{%- else %}
|
||||
# Patch edx-platform
|
||||
# Fix broken "Pages" view in Studio
|
||||
# https://github.com/openedx/edx-platform/pull/30550
|
||||
RUN curl -fsSL https://github.com/open-craft/edx-platform/commit/3d54f284f82b61e693ad652d8d6e46a226fcb36d.patch | git am
|
||||
# Fix xblock ajax handler vulnerability
|
||||
# https://github.com/overhangio/edx-platform/tree/overhangio/sec-xblock-ajax
|
||||
RUN curl -fsSL https://github.com/overhangio/edx-platform/commit/3f0f9eed42.patch | git am
|
||||
# Fix XSS vulnerability on "next" parameter
|
||||
# https://github.com/overhangio/edx-platform/tree/overhangio/sec-redirect-xss
|
||||
RUN curl -fsSL https://github.com/overhangio/edx-platform/commit/e16f8c0986.patch | git am
|
||||
# Fix drag-n-drop v2 xblock vulnerability
|
||||
# https://github.com/openedx/edx-platform/pull/31354
|
||||
RUN curl -fsSL https://github.com/overhangio/edx-platform/commit/527b4993ae.patch | git am
|
||||
# Fix refresh of JWT tokens for disabled users
|
||||
# https://github.com/overhangio/edx-platform/tree/overhangio/sec-fix-user-retiring
|
||||
RUN curl -fsSL https://github.com/overhangio/edx-platform/commit/4df942b32b.patch | git am
|
||||
{%- endif %}
|
||||
|
||||
{# Example: RUN curl -fsSL https://github.com/openedx/edx-platform/commit/<GITSHA1> | git am #}
|
||||
|
|
@ -84,12 +69,6 @@ ENV VIRTUAL_ENV /openedx/venv/
|
|||
|
||||
RUN apt update && apt install -y software-properties-common libmysqlclient-dev libxmlsec1-dev libgeos-dev
|
||||
|
||||
# Note that this means that we need to reinstall all requirements whenever there is a
|
||||
# change in edx-platform, which sucks. But there is no obvious alternative, as we need
|
||||
# to install some packages from edx-platform.
|
||||
COPY --from=code /openedx/edx-platform /openedx/edx-platform
|
||||
WORKDIR /openedx/edx-platform
|
||||
|
||||
# Install the right version of pip/setuptools
|
||||
# https://pypi.org/project/setuptools/
|
||||
# https://pypi.org/project/pip/
|
||||
|
|
@ -97,7 +76,8 @@ WORKDIR /openedx/edx-platform
|
|||
RUN pip install setuptools==62.1.0 pip==22.0.4 wheel==0.37.1
|
||||
|
||||
# Install base requirements
|
||||
RUN pip install -r ./requirements/edx/base.txt
|
||||
COPY --from=code /openedx/edx-platform/requirements/edx/base.txt /tmp/base.txt
|
||||
RUN pip install -r /tmp/base.txt
|
||||
|
||||
# Install django-redis for using redis as a django cache
|
||||
# https://pypi.org/project/django-redis/
|
||||
|
|
@ -124,14 +104,14 @@ ENV PATH /openedx/nodeenv/bin:/openedx/venv/bin:${PATH}
|
|||
|
||||
# Install nodeenv with the version provided by edx-platform
|
||||
RUN pip install nodeenv==1.6.0
|
||||
RUN nodeenv /openedx/nodeenv --node=12.13.0 --prebuilt
|
||||
RUN nodeenv /openedx/nodeenv --node=16.14.0 --prebuilt
|
||||
|
||||
# Install nodejs requirements
|
||||
ARG NPM_REGISTRY={{ NPM_REGISTRY }}
|
||||
COPY --from=code /openedx/edx-platform/package.json /openedx/edx-platform/package.json
|
||||
COPY --from=code /openedx/edx-platform/package-lock.json /openedx/edx-platform/package-lock.json
|
||||
WORKDIR /openedx/edx-platform
|
||||
RUN npm clean-install --verbose --registry=$NPM_REGISTRY
|
||||
RUN npm install --verbose --registry=$NPM_REGISTRY
|
||||
|
||||
###### Production image with system and python requirements
|
||||
FROM minimal as production
|
||||
|
|
@ -161,8 +141,9 @@ ENV PATH /openedx/venv/bin:./node_modules/.bin:/openedx/nodeenv/bin:${PATH}
|
|||
ENV VIRTUAL_ENV /openedx/venv/
|
||||
WORKDIR /openedx/edx-platform
|
||||
|
||||
# Re-install local requirements, otherwise egg-info folders are missing
|
||||
RUN pip install -r requirements/edx/local.in
|
||||
# We install edx-platform here because it creates an egg-info folder in the current
|
||||
# repo. We need both the source code and the virtualenv to run this command.
|
||||
RUN pip install -e .
|
||||
|
||||
# Create folder that will store lms/cms.env.yml files, as well as
|
||||
# the tutor-specific settings files.
|
||||
|
|
|
|||
|
|
@ -52,7 +52,7 @@ OPENEDX_LMS_UWSGI_WORKERS: 2
|
|||
OPENEDX_MYSQL_DATABASE: "openedx"
|
||||
OPENEDX_CSMH_MYSQL_DATABASE: "{{ OPENEDX_MYSQL_DATABASE }}_csmh"
|
||||
OPENEDX_MYSQL_USERNAME: "openedx"
|
||||
OPENEDX_COMMON_VERSION: "open-release/nutmeg.2"
|
||||
OPENEDX_COMMON_VERSION: "master"
|
||||
OPENEDX_EXTRA_PIP_REQUIREMENTS:
|
||||
- "openedx-scorm-xblock<15.0.0,>=14.0.0"
|
||||
MYSQL_HOST: "mysql"
|
||||
|
|
|
|||
|
|
@ -1,8 +0,0 @@
|
|||
echo "Loading settings $DJANGO_SETTINGS_MODULE"
|
||||
|
||||
# Import demo course
|
||||
git clone https://github.com/openedx/edx-demo-course --branch {{ OPENEDX_COMMON_VERSION }} --depth 1 ../edx-demo-course
|
||||
python ./manage.py cms import ../data ../edx-demo-course
|
||||
|
||||
# Re-index courses
|
||||
./manage.py cms reindex_course --all --setup
|
||||
|
|
@ -203,6 +203,10 @@ def is_a_tty() -> bool:
|
|||
|
||||
def execute(*command: str) -> int:
|
||||
click.echo(fmt.command(_shlex_join(*command)))
|
||||
return execute_silent(*command)
|
||||
|
||||
|
||||
def execute_silent(*command: str) -> int:
|
||||
with subprocess.Popen(command) as p:
|
||||
try:
|
||||
result = p.wait(timeout=None)
|
||||
|
|
|
|||
Loading…
Reference in New Issue