6
0
mirror of https://github.com/ChristianLight/tutor.git synced 2024-12-14 06:58:21 +00:00
Commit Graph

63 Commits

Author SHA1 Message Date
Régis Behmo
0c1d4ebf51 fix: https test 2021-12-20 21:19:10 +01:00
Régis Behmo
b651ebc3c4 fix: segmentation fault on Mac OS M1 (Big Sur)
On Mac OS M1 (ARM processor), when the GMP library is available, pycryptodome
crashes with a segmentation fault during the generation of the RSA keys.

Upstream issue: https://github.com/Legrandin/pycryptodome/issues/506
Upstream fix: https://github.com/Legrandin/pycryptodome/pull/541

Close #473.
2021-09-17 10:44:52 +02:00
Régis Behmo
ef189e7f67 fix: better logging during plugins disable
When disable a plugin that set config entried, such as the minio plugin, tutor was logging the following:

    Disabling plugin minio...
        Removed config entry OPENEDX_AWS_ACCESS_KEY=openedx
        Removed config entry OPENEDX_AWS_SECRET_ACCESS_KEY={{ MINIO_AWS_SECRET_ACCESS_KEY }}
        Plugin disabled

The config values were not rendered during printing, which is a shame, because
the whole point of this log line is to warn users of passwords/secrets that are
being removed. Here, we make sure that the config values are properly rendered.
The new logs are now:

    Disabling plugin minio...
        Removing config entry OPENEDX_AWS_ACCESS_KEY=openedx
        Removing config entry OPENEDX_AWS_SECRET_ACCESS_KEY=64vpCVLxhDxBuNjakSrX4CQg
        Plugin disabled
2021-06-22 12:28:58 +02:00
Régis Behmo
ceddc11c29 feat: upgrade to open-release/lilac.master
One of the breaking changes of this release is the removal of the webui and
android features; these are moved to dedicated plugins. This causes a breaking
change: the renaming of the DOCKER_IMAGE_ANDROID
config variable to ANDROID_DOCKER_IMAGE.

See this TEP for reference: https://discuss.overhang.io/t/separate-webui-and-android-from-tutor-core-and-move-to-dedicated-plugins/1473
2021-06-08 23:29:12 +02:00
Régis Behmo
e990291d16 feat: upgrade pinned requirements to click 8+
We were forced to pin click to < v8 because of missing dependencies. In
particular, click_repl was broken. This is no longer the case, as click_repl
0.20 was published. Also, Jinja2 now includes type annotations, which allows us
to get rid of a few "# type: ignore" statements.

We take the opportunity to upgrade all requirements, which allows us resolve a
security issue on urllib3<1.26.0.
2021-06-06 14:38:59 +02:00
Régis Behmo
6d92fe2d4a fix: crash during local quickstart -p
When running `tutor local quickstart -p` we were getting the following error:

    Usage: custom [OPTIONS] ARGS...
    Try 'custom --help' for help.

    Error: Missing argument 'ARGS...'.

The docker-compose command sometimes accept a single command ("pull") with zero
argument.

See: https://discuss.overhang.io/t/local-quickstart-not-working-when-pullimages-enabled/1526
2021-05-07 17:01:09 +02:00
Eric Herrera
389dd96fdd fix: YamlParamType supports line terminators
This fix allows using a multiple line formatted Yaml string as input for setting a Tutor config value.
2021-04-25 09:14:21 +02:00
Régis Behmo
336cb79fa8 refactor: better config type checking
I stumbled upon a bug that should have been detected by the type
checking. Turns out, considering that config is of type Dict[str, Any]
means that we can use just any method on all config values -- which is
terrible. I discovered this after I set `config["PLUGINS"] = None`:
this triggered a crash when I enabled a plugin.
We resolve this by making the Config type more explicit. We also take
the opportunity to remove a few cast statements.
2021-04-18 18:02:02 +02:00
Régis Behmo
c01f4476b8 fix: TypeError when PLUGINS is None
When the PLUGINS config entry is None (`PLUGINS:`), the following error
was being triggered:

  File "/.../tutor/tutor/plugins.py",
  line 304, in is_enabled
      return name in config.get(CONFIG_KEY, [])
      TypeError: argument of type 'NoneType' is not iterable
2021-03-30 09:23:16 +02:00
Régis Behmo
0a670d7ead refactor: add type annotations
Annotations were generated with pyannotate:
https://github.com/dropbox/pyannotate

We are running in strict mode, which is awesome!

This affects a large part of the code base, which might be an issue for
people running a fork of Tutor. Nonetheless, the behavior should not be
affected. If anything, this process has helped find and resolve a few
type-related bugs. Thus, this is not considered as a breaking change.
2021-03-15 21:46:55 +01:00
Régis Behmo
070b75caa4 fix: parsing of YAML config params with "=" signs
Close #405.
2021-03-08 12:24:40 +01:00
Régis Behmo
41bb6fe190 minor: test file formatting 2021-02-25 10:03:45 +01:00
Régis Behmo
de872282b2 fix: minor edge case in long_to_base64 utility function.
This was detected by mypy.
2021-02-25 09:54:46 +01:00
Régis Behmo
1c927c6e96 Automatically bind-mount volumes from volumes/
This introduces a new dev/local command:

    tutor dev bindmount CONTAINER PATH

And a new volume syntax:

    tutor dev run --volume=PATH CONTAINER

This syntax automatically bind-mounts folders from the tutorroot/volumes
directory, which is pretty nifty.
2021-01-12 22:43:06 +01:00
Régis Behmo
728ef966dc v11.0.0 (2020-12-09)
- 💥[Improvement] Upgrade Open edX to Koa
- 💥 Setting changes:
    - The ``ACTIVATE_HTTPS`` setting was renamed to ``ENABLE_HTTPS``.
    - Other ``ACTIVATE_*`` variables were all renamed to ``RUN_*``.
    - The ``WEB_PROXY`` setting was removed and ``RUN_CADDY`` was added.
    - The ``NGINX_HTTPS_PORT`` setting is deprecated.
- Architectural changes:
    - Use Caddy as a web proxy for automated SSL/TLS certificate generation:
	- Nginx no longer listens to port 443 for https traffic
	- The Caddy configuration file comes with a new ``caddyfile`` patch for much simpler SSL/TLS management.
	- Configuration files for web proxies are no longer provided.
	- Kubernetes deployment no longer requires setting up a custom Ingress resource or custom manager.
    - Gunicorn and Whitenoise are replaced by uwsgi: this increases boostrap performance and makes it no longer necessary to mount media folders in the Nginx container.
    - Replace memcached and rabbitmq by redis.
- Additional features:
    - Make it possible to disable all plugins at once with ``plugins disable all``.
    - Add ``tutor k8s wait`` command to wait for a pod to become ready
    - Faster, more reliable static assets with local memory caching
- Deprecation: proxy files for Apache and Nginx are no longer provided out of the box.
- Removed plugin `{{ patch (...) }}` statements:
    - "https-create", "k8s-ingress-rules", "k8s-ingress-tls-hosts": these are no longer necessary. Instead, declare your app in the "caddyfile" patch.
    - "local-docker-compose-nginx-volumes": this patch was primarily used to serve media assets. The recommended is now to serve assets with uwsgi.
2020-12-10 01:05:02 +01:00
Régis Behmo
998dec7149 Fix template name separator for Windows users
See: https://github.com/overhangio/tutor/issues/381
2020-11-07 16:37:43 +01:00
Régis Behmo
7ed49e4716 Fix formatting according to newest black release 2020-10-15 17:33:28 +02:00
Régis Behmo
facc0b84e1 Make tutor considerably faster
Tutor was making many calls to iter_installed (~100 on my machine with a
dozen installed plugins). Turns out it's useless to cache Plugin and
Renderer instances, as the config keeps changing all the time. Instead,
we cache the list of installed plugins, which does not change in the
course of a single run.

On my machine this speeds up `tutor config save` by 5x, going from 7.5s
to 1.3s.
2020-10-15 16:28:55 +02:00
Régis Behmo
f5f501dad0 Remove "set" config entries on disabling plugin
Close #241
2020-09-18 13:21:15 +02:00
Régis Behmo
5f81aae616 Fix images.get_tag unit test 2020-07-21 09:28:44 +02:00
Régis Behmo
4d6de0138a v10.0.0 Upgrade to Juniper (2020-06-15)
Here, we upgrade the Open edX platform from Ironwood to Juniper. This
upgrade does not come with many feature changes, but there are many
technical improvements under the hood:

- Upgrade from Python 2.7 to 3.5
- Upgrade from Mongodb v3.2 to v3.6
- Upgrade Ruby to 2.5.7

We took the opportunity to completely rething the way locally running
platforms should be accessed for testing purposes. It is no longer
possible to access a running platform from http://localhost and
http://studio.localhost. Instead, users should access
http://local.overhang.io and https://studio.local.overhang.io. This
drastically simplifies internal communication between Docker containers.

To upgrade, users should simply run:

    tutor local quickstart

For Kubernetes platform, the upgrade process is outlined when running:

    tutor k8s upgrade --from=ironwood
2020-06-15 10:19:07 +02:00
Régis Behmo
1b1d21c7e7 Fix unit tests in the presence of local yaml plugins 2020-04-29 16:15:10 +02:00
Régis Behmo
bce6432d85 Improve job running in local and k8s
Running jobs was previously done with "exec". This was because it
allowed us to avoid copying too much container specification information
from the docker-compose/deployments files to the jobs files. However,
this was limiting:

- In order to run a job, the corresponding container had to be running.
This was particularly painful in Kubernetes, where containers are
crashing as long as migrations are not correctly run.
- Containers in which we need to run jobs needed to be present in the
docker-compose/deployments files. This is unnecessary, for example when
mysql is disabled, or in the case of the certbot container.

Now, we create dedicated jobs files, both for local and k8s deployment.
This introduces a little redundancy, but not too much. Note that
dependent containers are not listed in the docker-compose.jobs.yml file,
so an actual platform is still supposed to be running when we launch the
jobs.

This also introduces a subtle change: now, jobs go through the container
entrypoint prior to running. This is probably a good thing, as it will
avoid forgetting about incorrect environment variables.

In k8s, we find ourselves interacting way too much with the kubectl
utility. Parsing output from the CLI is a pain. So we need to switch to
the native kubernetes client library.
2020-04-25 23:12:42 +02:00
Régis Behmo
411327662e Add encrypt template filter
This is convenient for htpasswd-based authentication to nginx, for
instance.
2020-04-04 18:22:15 +02:00
Régis Behmo
7eb99a3811 Minor code refactoring, for naming clarity 2020-02-27 17:14:00 +01:00
Régis Behmo
731de6a747 Fix incorrectly parsed empty strings in config save --set ...=...
When running `config save --set KEY=""`, KEY was incorrectly interpreted
as null.
2020-01-21 18:32:34 +01:00
Régis Behmo
2159226b4b Fix missing partials folder in config render
"partials" folders were not rendered at all when using `config render`.
2020-01-16 23:58:20 +01:00
Régis Behmo
72e23f3f96 Add config render command
This is going to be useful for using custom themes with user-defined
variables.
2020-01-16 17:15:55 +01:00
Régis Behmo
c4c12b0ab8 env.py refactoring
Clarify a few variable names, make code more modular. Also, the Renderer
class now makes more sense as a singleton. We took the opportunity to
delete quite a lot of code.
2020-01-16 15:07:35 +01:00
Régis Behmo
e3444d668c Add support for simple, yaml-based plugins
Those plugins are stored as yaml files in ~/.local/share/tutor-plugins
and follow the same specifications as entrypoint plugins.
2020-01-14 17:35:51 +01:00
Régis Behmo
159e24a95d Refactor plugin internals
This is for supporting json-based plugins. The great thing about this
change is that it allows us to easily print plugin version numbers in
`plugins list`.
2020-01-14 15:41:42 +01:00
Régis Behmo
c1e47cec82 Minor internal function renaming 2020-01-14 12:01:06 +01:00
Régis Behmo
993694909a Fix forum depends_on when data services are disabled
When both mongodb and elasticsearch were not activated, there was a
syntax error in the local docker-compose.yml file.

Close #266
2019-11-22 09:21:59 +01:00
Régis Behmo
b01f4d9c0e Better dev environment
The `dev` commands now rely on a different openedx-dev docker image.
This gives us multiple improvements:

- no more chown in base image
- faster chown in development
- mounted requirements volume in development
- fix static assets issues
- bundled ipdb/vim/... packages, which are convenient for development

Close #235
2019-10-24 20:03:36 +02:00
Régis Behmo
0c0db78310 Get rid of mysql-client container
This has an impact on plugin hooks. Plugin hooks that needed to run
inside mysql-client now need to run inside mysql container. This
simplifies the deployment, as we no longer have an empty mysql-client
container sitting around.

When mysql is not enabled (ACTIVATE_MYSQL=False) the mysql container is
simply a mysql client.
2019-09-03 10:36:10 +02:00
Régis Behmo
2c01c8cc01 Fox test_env formatting 2019-08-21 18:41:55 +02:00
Régis Behmo
7790028cf7 Move tutor-minio to dedicated plugin repo 2019-08-20 17:46:53 +02:00
Régis Behmo
d4a429bcc5 Fix parsing of weird characters in configuration parameters 2019-07-11 10:31:23 +08:00
Régis Behmo
1e995aaf1a More robust plugin/env config checking 2019-07-10 16:20:43 +08:00
Régis Behmo
e33b61fd04 Improve command line yaml value deserialization 2019-07-10 15:22:12 +08:00
Régis Behmo
11e735f4e5 Migrate notes to a dedicated plugin 2019-07-04 09:31:12 +08:00
Régis Behmo
07a0323d8e Move Xqueue to a dedicated plugin
This gives us the opportunity to develop new hooks: build-image and
remote-image.
2019-07-04 09:31:12 +08:00
Régis Behmo
1f69e67b9f Fix missing password values from generated configuration
Passwords were not being stored to config.yml during the first
quickstart.
2019-06-24 12:36:19 +02:00
Régis Behmo
84f2060d33 Working Kubernetes quickstart
The k8s quickstart command is now functional, with suppport for https,
xqueue, notes and minio. There are still a few bugs to get rid of,
though.
2019-06-17 22:38:55 +02:00
Régis Behmo
9bd6197cd1 Fix a few minor plugin-related TODOs 2019-06-07 22:49:45 +02:00
Régis Behmo
a0146de611 Minor formatting and fix tests 2019-06-07 22:49:45 +02:00
Régis Behmo
6a68c4cc20 Progress on the plugins/k8s front
This commit introduces many changes:
- a fully functional minio plugin for local installation
- an almost-functional native k8s deployment
- a new way to process configuration, better suited to plugins

There are still many things to do:
- get rid of all the TODOs
- get a fully functional minio plugin for k8s
- add documentation for pluginso
- ...
2019-06-07 22:49:45 +02:00
Régis Behmo
f5c225231f Move config logic to dedicated non-command module 2019-06-07 22:49:45 +02:00
Régis Behmo
334f3e720e Towards a stable Kubernetes integration
Missing features:
- https certificates
- xqueue
- lms/cms workers

Moreover, we scalability issues due to the uploaded file storage in the
lms/cms. To address this issue we need to develop the MinIO plugin so
that it becomes compatible with Open edX.

Close #126 #179 #187
2019-06-07 22:49:45 +02:00
Régis Behmo
3b108d21de 🔌 Introduce a plugin system for tutor
This adds the basic feratures that we need for a working plugin system,
but there are still many TODOs in the codebase.
2019-06-07 22:49:45 +02:00