There was an issue with non ascii platform names, such as "Курсы". This
is fixed by switching to utf8-encode settings file. That is, until we
switch to python3.
Close #270
It makes more sense to document this command instead of adding it to the
`local` commands. If need be, in the future we should be able to re-add
it as a plugin.
This command adds a burden on the `local` and `k8s` command. It does not
make sense to provide this command out of the box, and not other
administration commands. Instead, we should better document how to run
regular `manage.py` commands from tutor.
Close #269.
- [Improvement] Add `k8s-deployments-nginx-volume-mounts` patch
- [Bugfix] Fix running forum locally when both elasticsearch and mongodb
are not activated (#266)
- [Bugfix] Fix MongoDb url in forum when running separate service (#267)
- 💥[Improvement] Better `dev` commands, with dedicated development
docker image. One of the consequences is that the `dev watchthemes`
command is replaced by `dev run lms watchthemes`.
- [Improvement] `images` commands now accept multiple `image` arguments
MONGOHQ_URL was not properly set when MONGODB_HOST/PORT was customised.
That's because the environment variable was being defined in the
Dockerfile, and not at runtime.
Close #267
The static assets need to be properly collected in the development image
in order to have a working webpack-stats.json file.
The /openedx/staticfiles folder is still bind-mounted in the docker
image in order to have a working `watchthemes` command.
This is for issue #235, again.
The `dev` commands now rely on a different openedx-dev docker image.
This gives us multiple improvements:
- no more chown in base image
- faster chown in development
- mounted requirements volume in development
- fix static assets issues
- bundled ipdb/vim/... packages, which are convenient for development
Close #235
Use "{% include %}" jinja directive instead of python "exec", which was
a bit weird.
Also, this sets the technical email for the studio, which was previous
set at "technical@example.com".
- [Feature] Introduce tutor docker image
- [Feature] Introduce `local hook` command for plugin development.
- [Bugfix] Persist `private.txt` file between two runs of `config save`.
(#247)
- [Improvement] Added configuration values to limit the number of
gunicorn workers for the LMS and CMS.
- 💥[Improvement] Get rid of mysql-client container
- [Improvement] Add "local-docker-compose-lms/cms-dependencies" plugin
patches
- [Improvement] Use "exec" instead of "run" to initialise local platform
This has an impact on plugin hooks. Plugin hooks that needed to run
inside mysql-client now need to run inside mysql container. This
simplifies the deployment, as we no longer have an empty mysql-client
container sitting around.
When mysql is not enabled (ACTIVATE_MYSQL=False) the mysql container is
simply a mysql client.
We ran into an issue when trying to run migrations when the MinIO plugin
is activated. As seen in issues #243 and #244, the
certificates.0003_data__default_modes migration requires access to
MinIO. To do so, the MinIO host must be reached. That means that SSL
certificates must be in place (if https is enabled) and that the nginx
server must be booted. However, it does not make sense to require that
the minio container depends on the nginx container. So, in effect, we
need a fully working platform to run migrations.
In a sense, this is better as it harmonises the init task with k8s: in
k8s, init was already run with exec.
Next step is to get rid of these ugly mysql-client/minio-client
containers that must be up at all times. It would be much simpler to
just exec the commands inside the mysql/minio containers.
- [Feature] Modify ``createuser`` commands to define a password from the
command line
- [Improvement] Better yaml value parsing from command line
- [Feature] Add `dev exec` command
- [Bugfix] Fix incorrect notes settings definition
- [Improvement] Make it possible to start/stop/reboot a selection of
services
- [Improvement] Add `local/k8s reboot` commands
- [Improvement] Add `-U/--unset` option to `config save`
- [Bugfix] Fix insecure static asset loading when web proxy is enabled
- [Improvement] Rename `SECRET_KEY` configuration parameter to
`OPENEDX_SECRET_KEY`
- [Improvement] Add support for SSL and TLS in external SMTP server
(#231)
- [Bugfix] Fix missing video transcripts in LMS (#229)
- [Improvement] Make it possible to enable/disable multiple plugins at
once
- [Improvement] Add a few local and k8s patches for plugins
We have a side effect whenever we add a plugin manually to the tutor
configuration: the renderer does not update itself. We fix this by
checking the config at load time.
We achieve this thanks to a patch to edx-oauth2-provider:
https://github.com/edx/edx-oauth2-provider/pull/59
The network alias was required to circumvent url validation by the
oauth2 client creation tool. But the network alias was not compatible
with k8s deployment.
The port number is not included in the hostname when checking the list
of allowed values.
We also add the "lms"/"cms" values for interaction with the API from
within the internal docker network.
Video transcripts uploaded in the CMS were not visible in the LMS. This
was a symptom caused by the fact that the LMS and the CMS do not share
the same MEDIA_ROOT. We initially thought that data uploaded in the CMS
(such as transcripts) was stored in a shared data service, such as
mongodb. It is, in fact, not. This makes it even more important to run
an object storage service like minio for distributed services.
Close #229
The 0003 migration from the certificates app of the LMS requires that
the S3-like platform is correctly setup during initialisation. To solve
this issue, we introduce a pre-init hook that is run prior to the LMS
migrations.
P3P is obsolete https://en.wikipedia.org/wiki/P3P
Also, these P3P headers leak information about the application that is
running, making it sensitive to attacks.
- [Feature] Creation of a plugin system
- [Feature] Kubernetes support out of beta
- [Improvement] Switch to pinned image tags for easier upgrades
- 💥[Improvement] Remove the `-y/--yes` option: `tutor config save` is
now non-interactive by default. Use `-i/--interactive` to force
interactive mode.
- 💥[Improvement] Replace the `databases` command by `init`.
- [Improvement] Upgrade to ironwood.2
- [Improvement] Add `-y/--yes` option to `local quickstart` for
non-interactive quickstart
- [Improvement] Persist LMS/CMS logs to disk by default (with
collaboration from @silviot 💪)
- [Bugfix] Fix installing a locally cloned requirement repository
- [Improvement] Add `--no-cache` option to `images build`
- [Improvement] Make it possible to configure the notes service hostname
- [Improvement] Better, more robust MySQL initialisation
Having an identical "ironwood" tag for all releases is not practical, in
particular for breaking changes. Thus, docker images are now pinned to
the tutor version that they were build with.
Thus, we remove the -y/--yes options, which were kind of unintuitive,
and we add instead `-i/--interactive`. The quickstart commands remain
interactive by default, but can be silenced with `-I/--non-interactive`.
This commit introduces many changes:
- a fully functional minio plugin for local installation
- an almost-functional native k8s deployment
- a new way to process configuration, better suited to plugins
There are still many things to do:
- get rid of all the TODOs
- get a fully functional minio plugin for k8s
- add documentation for pluginso
- ...
Missing features:
- https certificates
- xqueue
- lms/cms workers
Moreover, we scalability issues due to the uploaded file storage in the
lms/cms. To address this issue we need to develop the MinIO plugin so
that it becomes compatible with Open edX.
Close #126#179#187
- [Improvement] Upgrade to ironwood.2
- [Improvement] Add `-y/--yes` option to `local quickstart` for non-interactive quickstart
- [Improvement] Persist LMS/CMS logs to disk by default (with collaboration from @silviot 💪)
- [Bugfix] Fix installing a locally cloned requirement repository
- [Improvement] Add `--no-cache` option to `images build`
- [Improvement] Make it possible to configure the notes service hostname
- [Improvement] Better, more robust MySQL initialization
This is especially useful on Kubernetes. With this change, the forum
container no longer crashes whenever mongodb or elasticsearch are
unavailable. Instead, it just waits for thoses services to be up.
Previously, we could not run forum migrations in Kubernetes because they
relied on exec-ing the migration command in the running container -- and
there was no such container, because the services where not already up.
Configuration loading was overly complex. Here, we simplify it
drastically with reasonable defaults. Hacky additional variables are
unncessary now that we use custom jinja2 filters.
For initializing the mysql database, we now run the mysql client from
the mysql container. That means that the mysql client is no longer
required in the openedx container, as previously suggested in PR #209.
When running `pip install -r /path/to/x.txt`, relative paths listed in
x.txt are supposed to be relative to the current folder, and not the
x.txt folder. This results in an error during `pip install`:
./xblock-poll/ should either be a path to a local project or a VCS
url beginning with svn+, git+, hg+, or bzr+
This error was first reported here:
https://discuss.overhang.io/t/installing-custom-xblock/61
Tutor is not using any bash-specific feature.
This commit changes (almost) all invocations to /bin/bash replacing them with
/bin/sh. It allows building a docker image that does not bundle bash in it.
The file k8s.py still has a bash invocation, to make sure this change does not
impact functionality for any user.
Given that each gunicorn worker uses ~200-250Mb virtual memory, we should
really avoid running too many of them. Here, we reduce the default
number of workers to 2. For the LMS and the CMS, this number can be
adjusted by setting an environment variable GUNICORN_WORKERS.
This is related to #207.
Settings the number of workers for the LMS and the CMS to 2*|CPU|+1
results in 4*|CPU]+2 gunicorn workers, which is too much for a laptop to
cope with.
Close #207
Migrations were being done on openedx-assets, which does not depend on
mysql. As a consequence, the mysql container was not being created on
"migrate" and migrations were failing with "Timeout after 20s".
