6
0
mirror of https://github.com/ChristianLight/tutor.git synced 2024-12-12 14:17:46 +00:00
Commit Graph

125 Commits

Author SHA1 Message Date
Overhang.IO
71e469174f Merge remote-tracking branch 'origin/master' into nightly 2023-12-10 22:19:14 +00:00
Régis Behmo
0d997c9479 feat: auto-mount edx-platform python requirements
These changes make to possible to run:

    tutor mounts add /path/to/my-xblock

The xblock directory with then be auto-magically bind-mounted in the
"openedx" image at build time, and the lms*/cms* containers at run time.

This makes it effectively possible to work as a developer on
edx-platform requirements.

We take the opportunity to move some openedx-specific code to a
dedicated module.

Close https://github.com/openedx/wg-developer-experience/issues/177
2023-12-10 22:57:08 +01:00
Emad Rad
61d37b436d fix: save configs by enable/disable plugins.
before this, after enabling/disabling any plugins we should re-generate all files with tutor config save.
2023-12-08 12:47:34 +01:00
Régis Behmo
913a1adae4 Merge branch 'master' into nightly 2023-11-23 12:58:04 +01:00
Omar Al-Ithawi
65ba0d2de2 fix: fix ulimits error for elasticsearch in Docker rootless mode
disable `ulimits` in rootless docker mode by setting them to zero
2023-11-23 12:36:07 +01:00
Kyle McCormick
c273e7f105
feat!: assume BuildKit is available 2023-10-27 10:54:23 +02:00
Régis Behmo
51928b0075 fix: parse strings prefixed with "#" in config save --set ...
Pound keys were interpreted as comments. This is annoying when we want
to parse html color codes, such as in:

    $ tutor config save --set "INDIGO_PRIMARY_COLOR=#225522"
    $ tutor config printvalue INDIGO_PRIMARY_COLOR
    null

Close #866.
2023-09-06 12:06:10 +02:00
Régis Behmo
8eccaa61e9 fix: render list config items
Close #867.
2023-09-05 16:06:44 +02:00
Régis Behmo
2a47100d6a fix: broken mysql after palm upgrade
This fix is for a rather serious issue that affects users who upgrade
from Olive to Palm. The client mysql charset and collation was
incorrectly set to utf8mb4, while the server stil runs utf8mb3. Only
users who run the mysql container are affected.

To resolve this issue, we explicitely configure the client to use the
utf8mb3 charset/collation.

Important note: users who have somehow managed to upgrade from olive to
Palm before may find themselves in an undefined state. They might have
to fix their mysql data manually. Same thing for users who launched Palm
from scratch; although, according to my preliinary tests, they should be
able to downgrade their connection from utf8mb4 to utf8mb3 without
issue.

In addition, we upgrade to mysql 8.1.0. Among many other fixes, this
avoids a server restart after the upgrade:

> An in-place upgrade from MySQL 5.7 to MySQL 8.0, without a server
> restart, could result in unexpected errors when executing queries on
> tables. This fix eliminates the need to restart the server between the
> upgrade and queries. (Bug #35410528)

https://dev.mysql.com/doc/relnotes/mysql/8.0/en/news-8-0-34.html

See also the 8.1.0 release notes:

https://dev.mysql.com/doc/relnotes/mysql/8.1/en/news-8-1-0.html

Close #887.
2023-08-16 19:01:41 +02:00
Régis Behmo
67009fb857 fix: improve support of non-buildkit Docker build
See comment here:
https://github.com/overhangio/tutor/pull/868#issuecomment-1640429396

See also the conversation that spawned this PR:
https://discuss.openedx.org/t/issue-in-tutor-palm-release-with-tuotr-dev-launch-while-installing/10629
2023-08-16 18:56:59 +02:00
Régis Behmo
cac8530481 fix: format output of config printvalue as yaml 2023-06-14 21:08:49 +02:00
Régis Behmo
69944e4028 feat: separate mounts command
Manual configuration via the `MOUNTS` setting was inconvenient. We
(re)introduce a new(ish) `tutor mounts` command. Old timers will perhaps
remember that we used to have a `tutor bindmount` command. Well, it's
back! But better and different.
2023-06-14 21:08:49 +02:00
Régis Behmo
17f66fb467 refactor: simplify image tag management 2023-06-14 21:08:49 +02:00
Régis Behmo
3ab0dcb9e6 depr: templated hooks
Templated hooks we almost completely useless, so we get rid of them.
This allows us to get rid entirely of hook names and hook indexes, which
makes the whole implementation much simpler. Hook removal (with
`clear_all`) is achieved thanks to weak references.
2023-06-14 21:08:49 +02:00
Régis Behmo
18ce1f2fe4 feat: persistent bind-mounts
This is an important change, where we get remove the previous `--mount`
option, and instead opt for persistent bind-mounts.

Persistent bind mounts have several advantages:
- They make it easier to remember which folders need to be bind-mounted.
- Code is *much* less clunky, as we no longer need to generate temporary
  docker-compose files.
- They allow us to bind-mount host directories *at build time* using the
  buildx `--build-context` option.
- The transition from development to production becomes much easier, as
  images will automatically be built using the host repo.

The only drawback is that persistent bind-mounts are slightly less
portable: when a config.yml file is moved to a different folder, many
things will break if the repo is not checked out in the same path.

For instance, this is how to start working on a local fork of
edx-platform:

    tutor config save --append MOUNTS=/path/to/edx-platform

And that's all there is to it. No, this fork will be used whenever we
run:

    tutor images build openedx
    tutor local start
    tutor dev start

This change is made possible by huge improvements in the build time
performance. These improvements make it convenient to re-build Docker
images often.

Related issues:
https://github.com/openedx/wg-developer-experience/issues/71
https://github.com/openedx/wg-developer-experience/issues/66
https://github.com/openedx/wg-developer-experience/issues/166
2023-06-14 21:08:49 +02:00
Régis Behmo
7972a75915 feat: leverage RUN --mount for faster image building
We make use of the Docker build cache to install python and nodejs
requirements faster in the case of repeated builds.

This feature is only possible for users of BuildKit, so we detect
whether `docker buildx` is available at runtime.

We do not make use of `COPY --link` because the `--link` option is
incompatible with `--chown=app:app`:
https://github.com/docker/buildx/issues/1408

For reference, see:

https://www.docker.com/blog/dockerfiles-now-support-multiple-build-contexts/
https://docs.docker.com/engine/reference/commandline/buildx_build/#build-context
2023-06-14 21:08:49 +02:00
Régis Behmo
220b8296a9 feat: config save --append/--remove KEY=VAL options
This paves the way for persisting bind-mounts across runs, since this
setting will be a list.
2023-06-14 19:40:58 +02:00
Régis Behmo
b3c3c4a2cc feat: upgrade to Palm
Among other changes: ORA2 file uploads were stored in a folder named
"SET-ME-PLEASE (ex.  bucket-name)" (sigh). With this change, the folder
should be automatically renamed to "openedxuploads". This issue has been
occuring since June 2019... (sigh²)

Close #707
2023-06-14 19:40:58 +02:00
Régis Behmo
3377ffc38a fix: v0 plugin patch ordering
v0 plugin patches were directly added to ENV_PATCH, while v1 patches
were added to ENV_PATCHES. ENV_PATCHES items are forwarded to ENV_PATCH
later at runtime, which means that yaml plugin patches were always
loaded before v1 patches.

This was causing weird ordering issues, as discussed here:
https://discuss.openedx.org/t/using-mfe-config-to-set-logo-trademark-url-for-the-footer-component-in-olive/9578
2023-03-20 16:15:12 +01:00
Maria Fernanda Magallanes Zubillaga
f13627a32a feat: add cli for listing available patches 2023-03-09 13:26:33 +01:00
Maria Fernanda Magallanes Zubillaga
40e5c8da6b refactor: move format_table function to utils 2023-02-28 09:45:11 +01:00
Régis Behmo
fa318a64ce feat: plugin indexes
We implement this TEP: https://discuss.openedx.org/t/tutor-enhancement-proposal-tep-plugin-indices/8182

With plugin indexes, tutor users can install and upgrade plugins directly from indexes:

    tutor plugins install ecommerce
    tutor plugins index add contrib
    tutor plugins install codejail
    tutor plugins upgrade all

This change has been long in the coming \o/
2023-02-10 10:06:38 +01:00
Régis Behmo
7b718f6e78 chore: upgrade requirements
Cryptography in particular needs to be upgraded to 39.0.1:
https://github.com/overhangio/tutor/security/dependabot/7
https://github.com/overhangio/tutor/security/dependabot/8

Black upgrade caused minor spacing changes.
2023-02-09 12:07:09 +01:00
Régis Behmo
71ed7a8618 feat: refactor hooks API for simplification
The hooks API had several issues which are summarized in this comment:
https://github.com/openedx/wg-developer-experience/issues/125#issuecomment-1313553526

1. "consts" was a bad name
2. "hooks.filters" and "hooks.Filters" could easily be confused
3. docs made it difficult to understand that plugin developers should use the catalog

To address these issues, we:

1. move "consts.py" to "catalog.py"
2. Remove "hooks.actions", "hooks.filters", "hooks.contexts" from the API.
3. re-organize the docs and give better usage examples in the catalog.

This change is a partial fix for https://github.com/openedx/wg-developer-experience/issues/125
2023-01-31 10:17:58 +01:00
Carlos Muniz
ac1a875f42 refactor: annotation with __future__.annotations
Adds `from __future__ import annotations` to the top of every module,
right below the module's docstring. Replaces any usages of t.List,
t.Dict, t.Set, t.Tuple, and t.Type with their built-in equivalents:
list, dict, set, tuple, and type. Ensures that make test still passes
under Python 3.7, 3.8 and 3.9.
2023-01-18 08:36:14 +01:00
Régis Behmo
c26999ec65 feat: upgrade to olive 2022-12-12 16:55:07 +01:00
Overhang.IO
12fe99f699 Merge remote-tracking branch 'origin/master' into nightly 2022-11-29 15:07:54 +00:00
Régis Behmo
429d528d64 fix: unit test on github's macOs action runner
Somehow, the tmp directory is a symlink on github.
2022-11-29 16:02:17 +01:00
Overhang.IO
9466fc395b Merge remote-tracking branch 'origin/master' into nightly 2022-11-29 14:58:08 +00:00
Régis Behmo
3b6d2404e5 fix: unit tests when run as root
GitHub Actions sometimes runs `make tests` as root; e.g: in the release script.
There were unit tests that were breaking in that scenario. I have no idea why
tests were not breaking in the test.yml workflow.
2022-11-29 15:50:34 +01:00
Régis Behmo
85d868423a fix: do not prepend DJANGO settings info to all jobs
The DJANGO_SETTINGS_MODULE is far from being relevant in all containers.
2022-11-24 18:40:16 +01:00
Régis Behmo
c06ac5b020 Merge branch 'master' into nightly 2022-11-24 17:27:47 +01:00
Régis Behmo
29eb3398a2 feat: auto-complete config save/printroot arguments 2022-11-24 16:21:57 +01:00
Régis Behmo
ee09612326 feat: auto-complete plugins arguments
Support auto-completion of plugin name and path arguments in the `tutor
plugins` commands.
2022-11-24 16:21:57 +01:00
Régis Behmo
b4a8069cfd fix: init template paths
Plugin template paths of init jobs could not be found because they were
searched for in the Tutor template root only.
2022-11-21 10:03:25 +01:00
Régis Behmo
d7c667835a Merge branch 'master' into nightly 2022-11-15 16:59:19 +01:00
Régis Behmo
33e4f33afe feat: strongly typed hooks
Now that the mypy bugs have been resolved, we are able to define more precisely
and cleanly the types of Actions and Filters.

Moreover, can now strongly type named actions and hooks (in consts.py). With
such a strong typing, we get early alerts of hooks called with incorrect
arguments, which is nothing short of awesome :)

This change breaks the hooks API by removing the `context=...` argument. The
reason for that is that we cannot insert arbitrary arguments between `P.args,
P.kwargs`: https://peps.python.org/pep-0612/#the-components-of-a-paramspec

> A function declared as def inner(a: A, b: B, *args: P.args, **kwargs:
> P.kwargs) -> R has type Callable[Concatenate[A, B, P], R]. Placing
> keyword-only parameters between the *args and **kwargs is forbidden.

Getting the documentation to build in nitpicky mode is quite difficult... We
need to add `nitpick_ignore` to the docs conf.py, otherwise sphinx complains
about many missing class references. This, despite upgrading almost all doc
requirements (except docutils).
2022-11-15 14:58:36 +01:00
Régis Behmo
16e6131f96 feat: pluggable local/dev/k8s do <job> commands
We introduce a new filter to implement custom commands in arbitrary containers.
It becomes easy to write convenient ad-hoc commands that users will
then be able to run either on Kubernetes or locally using a documented CLI.

Pluggable jobs are declared as Click commands and are responsible for
parsing their own arguments. See the new CLI_DO_COMMANDS filter.

Close https://github.com/overhangio/2u-tutor-adoption/issues/75
2022-11-15 09:46:08 +01:00
Régis Behmo
b6dc65cc64 refactor: deduplicate jobs code
createuser, importdemocourse and settheme were 100% duplicated code between
k8s.py and compose.py.
2022-11-15 09:46:08 +01:00
Régis Behmo
e56918bf47 depr: get rid of the local/dev bindmount commands
This command has always been clunky. It is now removed in favour of the
`-m/--mount` option.

Close https://github.com/overhangio/2u-tutor-adoption/issues/88
Close https://github.com/overhangio/2u-tutor-adoption/issues/89
2022-10-19 17:51:06 +02:00
Carlos Muniz
fe901ab9de feat: deprecate "quickstart" and rename to "launch"
`quickstart` is being renamed to `launch` and deprecated in favor of
using `launch`. The `quickstart` function temporarily aliases to
`launch`. Further mentions of `quickstart` have been changed to
reference `launch` instead.

We are indicating that this change is breaking 💥 to encourage people to
migrate their scripts right away!
2022-09-30 12:05:35 +02:00
Régis Behmo
a2a3c022b8 fix: bind-mount in dev-specific services
The -m/--mount option makes it possible to bind-mount volumes at runtime. The
volumes are declared in a local/docker-compose.tmp.yml file. The problem with
this approach is when we want to bind-mount a volume to a service which is
specific to the dev context. For instance: the "learning" service when the MFE
plugin is enabled.

In such a case, starting the service triggers a call to `docker-compose stop`
in the local context. This call fails because the "learning" service does not
exist in the local context. Note that this issue only seems to occur with
docker-compose v1.

To resolve this issue, we create two additional filters for
the dev context, which emulate the behaviour of the local context. With this approach, we convert the -m/--mount arguments right after they are parsed. Because they are parsed just once, we can get rid of the de-duplication logic initially introduced with the COMPOSE_CLI_MOUNTS context.

Close #711. Close also https://github.com/overhangio/tutor-mfe/issues/57.
2022-07-29 19:53:02 +02:00
Kyle McCormick
6a3138152f fix: accurately log shell commands when running jobs
Whenever Tutor executes a shell command, it logs out said
command in order to aid in end user understanding/debugging.

In some cases (notably, when running jobs in containers)
the logged command was not accurately quoted. The command
was run correctly, because it was passed in pieces to
``subprocess.Popen``, which correctly joins the pieces together
into a valid POSIX shell command; however, the logged version
of the command was constructed by simply joining the pieces
with spaces. This usually works, but breaks down when running
complex shell commands with nested quoting.

This commit changes the logging to use ``shlex.join``, which
joins command pieces together in a POSIX-compliant way,
presumably the same way as ``subprocess.Popen``.

Example:

    tutor local importdemocourse

runs the shell command:

    docker-compose -f /home/kyle/.local/share/tutor/env/local/docker-compose.yml -f /home/kyle/.local/share/tutor/env/local/docker-compose.prod.yml -f /home/kyle/.local/share/tutor/env/local/docker-compose.tmp.yml --project-name tutor_local -f /home/kyle/.local/share/tutor/env/local/docker-compose.jobs.yml -f /home/kyle/.local/share/tutor/env/local/docker-compose.jobs.tmp.yml run --rm cms-job sh -e -c 'echo "Loading settings $DJANGO_SE... (several more script lines) ...eindex_course --all --setup'

but the logged shell command was:

    docker-compose -f /home/kyle/.local/share/tutor/env/local/docker-compose.yml -f /home/kyle/.local/share/tutor/env/local/docker-compose.prod.yml -f /home/kyle/.local/share/tutor/env/local/docker-compose.tmp.yml --project-name tutor_local -f /home/kyle/.local/share/tutor/env/local/docker-compose.jobs.yml -f /home/kyle/.local/share/tutor/env/local/docker-compose.jobs.tmp.yml run --rm cms-job sh -e -c echo "Loading settings $DJANGO_SE... (several more script lines) ...eindex_course --all --setup

which will not run if copied and pasted back into the
user's terminal, as the importdemocourse shell script is unquoted.
2022-07-25 22:46:53 +02:00
Kyle McCormick
0ae59a82a6
fix: avoid double-rendering mounts to docker-compose.tmp.yml (#669)
In certain code paths, such as in `tutor local quickstart`,
`process_mount_points` is called more than once in the same process,
causing mounts to be added to `COMPOSE_LOCAL[_JOBS]_TMP` redundantly.
As a result, docker-compose[.jobs].tmp.yml was occasionally being
rendered with duplicate volume specifiers. Some versions of Docker
Compose ignored this; other versions warned or threw an error.

In order to make `process_mount_points` tolerant to being called
multiple times, we wrap its volume-adding callbacks within a new
hooks context. This allows us to clear said hooks context every
time `process_mount_points` is called, essentially making the
function idempotent.

Co-authored-by: Régis Behmo <regis@behmo.com>
2022-07-25 15:58:49 +02:00
Régis Behmo
c4388e134c v14.0.0: upgrade to Nutmeg
- 💥 [Feature] Upgrade to Nutmeg: (by @regisb)
    - 💥 [Feature] Persistent grades are now enabled by default.
    - [Bugfix] Remove edX references from bulk emails ([issue](https://github.com/openedx/build-test-release-wg/issues/100)).
    - [Improvement] For Tutor Nightly (and only Nightly), official plugins are now installed from their nightly branches on GitHub instead of a version range on PyPI. This will allow Nightly users to install all official plugins by running ``pip install -e ".[full]"``.
    - [Bugfix] Start MongoDB when running migrations, because a new data migration fails if MongoDB is not running
2022-06-09 19:18:20 +02:00
Régis Behmo
82f2a448d2 feat: render files in ignored directories
When rendering theme files in a plugin, the *.scss files are stored in a
"partials" subdirectory, which was ignored by the environment rendering logic.
To render these files, we move the path ignoring logic to a filter, which is a
list of regular expressions. Values in this filter can be overridden by another
filter.

See the corresponding issue in the indigo theme plugin:
https://github.com/overhangio/tutor-indigo/issues/24
2022-06-03 12:54:27 +02:00
Régis Behmo
1e0c305508 fix: get rid of the tutor config render command
This command is useless now that we can implement themes as plugins. This
allows us to considerably simplify the Renderer class constructor.
2022-06-03 12:54:27 +02:00
Régis Behmo
b6ec87d17c fix: copyfrom unit test on python 3.7
`call_args.args` was only introduced in Python 3.8. This is breaking CI.
2022-04-24 10:39:51 +02:00
Régis Behmo
27449f4068 feat: add dev/local copyfrom commands
`copyfrom` copies data from a container to the local filesystem. It's similar
to bindmount, but less clunky, and more intuitive. Also, it plays along great
with `--mount`. Eventually we'll just get rid of the `bindmount` command and
the `--volume` option.
2022-04-24 09:51:46 +02:00
Kyle McCormick
126c6062e5 fix: --mount should accept service names with hyphens
(no changelog entry, as this fixes a feature that hasn't
 yet been released)
2022-04-22 10:14:11 +02:00