Tutor was making many calls to iter_installed (~100 on my machine with a
dozen installed plugins). Turns out it's useless to cache Plugin and
Renderer instances, as the config keeps changing all the time. Instead,
we cache the list of installed plugins, which does not change in the
course of a single run.
On my machine this speeds up `tutor config save` by 5x, going from 7.5s
to 1.3s.
Right now if I add .woff or .woff2 fonts files to an Indigo-based theme's `lms/static/fonts` directory, I get the following error:
```
$ /indigo-folder# make
tutor config render --extra-config ./config-totem.yml ./theme "$(tutor config printroot)/env/build/openedx/themes/indigo-totem"
Error loading template lms/static/fonts/NotoSans-Bold.woff
Traceback (most recent call last):
File "/home/maarten/.local/bin/tutor", line 8, in <module>
sys.exit(main())
File "/home/maarten/.local/lib/python3.8/site-packages/tutor/commands/cli.py", line 38, in main
cli() # pylint: disable=no-value-for-parameter
File "/usr/lib/python3/dist-packages/click/core.py", line 764, in __call__
return self.main(*args, **kwargs)
File "/usr/lib/python3/dist-packages/click/core.py", line 717, in main
rv = self.invoke(ctx)
File "/usr/lib/python3/dist-packages/click/core.py", line 1137, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/lib/python3/dist-packages/click/core.py", line 1137, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/lib/python3/dist-packages/click/core.py", line 956, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/lib/python3/dist-packages/click/core.py", line 555, in invoke
return callback(*args, **kwargs)
File "/usr/lib/python3/dist-packages/click/decorators.py", line 27, in new_func
return f(get_current_context().obj, *args, **kwargs)
File "/home/maarten/.local/lib/python3.8/site-packages/tutor/commands/config.py", line 86, in render
renderer.render_all_to(dst)
File "/home/maarten/.local/lib/python3.8/site-packages/tutor/env.py", line 153, in render_all_to
rendered = self.render_file(template)
File "/home/maarten/.local/lib/python3.8/site-packages/tutor/env.py", line 137, in render_file
template = self.environment.get_template(path)
File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 830, in get_template
return self._load_template(name, self.make_globals(globals))
File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 804, in _load_template
template = self.loader.load(self, name, globals)
File "/usr/lib/python3/dist-packages/jinja2/loaders.py", line 113, in load
source, filename, uptodate = self.get_source(environment, name)
File "/usr/lib/python3/dist-packages/jinja2/loaders.py", line 175, in get_source
contents = f.read().decode(self.encoding)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb4 in position 10: invalid start byte
make: *** [Makefile:2: render] Error 1
```
Here, we upgrade the Open edX platform from Ironwood to Juniper. This
upgrade does not come with many feature changes, but there are many
technical improvements under the hood:
- Upgrade from Python 2.7 to 3.5
- Upgrade from Mongodb v3.2 to v3.6
- Upgrade Ruby to 2.5.7
We took the opportunity to completely rething the way locally running
platforms should be accessed for testing purposes. It is no longer
possible to access a running platform from http://localhost and
http://studio.localhost. Instead, users should access
http://local.overhang.io and https://studio.local.overhang.io. This
drastically simplifies internal communication between Docker containers.
To upgrade, users should simply run:
tutor local quickstart
For Kubernetes platform, the upgrade process is outlined when running:
tutor k8s upgrade --from=ironwood
To do so, we had to create a new log_format.
Plugin developers are strongly encouraged to start using this log format
by adding the `access_log /var/log/nginx/access.log tutor;` directive to
their extra nginx configurations.
In order to load this log format early, the `tutor.conf` config file had
to be renamed to something early in the alphabet... (hence starting by
an underscore) Older users would face an error on nginx reload, so older
"tutor.conf" files are automatically removed on config save.
Clarify a few variable names, make code more modular. Also, the Renderer
class now makes more sense as a singleton. We took the opportunity to
delete quite a lot of code.
The `dev` commands now rely on a different openedx-dev docker image.
This gives us multiple improvements:
- no more chown in base image
- faster chown in development
- mounted requirements volume in development
- fix static assets issues
- bundled ipdb/vim/... packages, which are convenient for development
Close #235
We have a side effect whenever we add a plugin manually to the tutor
configuration: the renderer does not update itself. We fix this by
checking the config at load time.
Thus, we remove the -y/--yes options, which were kind of unintuitive,
and we add instead `-i/--interactive`. The quickstart commands remain
interactive by default, but can be silenced with `-I/--non-interactive`.
This commit introduces many changes:
- a fully functional minio plugin for local installation
- an almost-functional native k8s deployment
- a new way to process configuration, better suited to plugins
There are still many things to do:
- get rid of all the TODOs
- get a fully functional minio plugin for k8s
- add documentation for pluginso
- ...
Missing features:
- https certificates
- xqueue
- lms/cms workers
Moreover, we scalability issues due to the uploaded file storage in the
lms/cms. To address this issue we need to develop the MinIO plugin so
that it becomes compatible with Open edX.
Close #126#179#187
Configuration loading was overly complex. Here, we simplify it
drastically with reasonable defaults. Hacky additional variables are
unncessary now that we use custom jinja2 filters.
Installing from pip was broken because, for some reason, the
templates/apps/xqueue/settings/ folder contained a __pycache__ folder
with compiled *.pyc files. Rendering the binary files was failing
miserably.
Environment is no longer generated separately for each target, but only
once the configuration is saved.
Note that the environment is automatically updated during
re-configuration, based on a "version" file stored in the environment.
Replace all make commands by a single "tutor" binary. Environment and
data are all moved to ~/.tutor/local/share/tutor. We take the
opportunity to add a web UI and revamp the documentation.
This is a complete rewrite.
Close #121.
Close #147.
