"Course visibility In Catalog" advanced CMS setting was being ignored
because the default the COURSE_CATALOG_VISIBILITY_PERMISSION and
COURSE_ABOUT_VISIBILITY_PERMISSION settings were incorrectly set to the
legacy default, which "see_exists".
See
https://discuss.overhang.io/t/catalog-visibility-in-tutor-deployed-open-edx/214
There was an issue with non ascii platform names, such as "Курсы". This
is fixed by switching to utf8-encode settings file. That is, until we
switch to python3.
Close #270
It makes more sense to document this command instead of adding it to the
`local` commands. If need be, in the future we should be able to re-add
it as a plugin.
This command adds a burden on the `local` and `k8s` command. It does not
make sense to provide this command out of the box, and not other
administration commands. Instead, we should better document how to run
regular `manage.py` commands from tutor.
Close #269.
- [Improvement] Add `k8s-deployments-nginx-volume-mounts` patch
- [Bugfix] Fix running forum locally when both elasticsearch and mongodb
are not activated (#266)
- [Bugfix] Fix MongoDb url in forum when running separate service (#267)
- 💥[Improvement] Better `dev` commands, with dedicated development
docker image. One of the consequences is that the `dev watchthemes`
command is replaced by `dev run lms watchthemes`.
- [Improvement] `images` commands now accept multiple `image` arguments
MONGOHQ_URL was not properly set when MONGODB_HOST/PORT was customised.
That's because the environment variable was being defined in the
Dockerfile, and not at runtime.
Close #267
The static assets need to be properly collected in the development image
in order to have a working webpack-stats.json file.
The /openedx/staticfiles folder is still bind-mounted in the docker
image in order to have a working `watchthemes` command.
This is for issue #235, again.
The `dev` commands now rely on a different openedx-dev docker image.
This gives us multiple improvements:
- no more chown in base image
- faster chown in development
- mounted requirements volume in development
- fix static assets issues
- bundled ipdb/vim/... packages, which are convenient for development
Close #235
Use "{% include %}" jinja directive instead of python "exec", which was
a bit weird.
Also, this sets the technical email for the studio, which was previous
set at "technical@example.com".
- [Feature] Introduce tutor docker image
- [Feature] Introduce `local hook` command for plugin development.
- [Bugfix] Persist `private.txt` file between two runs of `config save`.
(#247)
- [Improvement] Added configuration values to limit the number of
gunicorn workers for the LMS and CMS.
- 💥[Improvement] Get rid of mysql-client container
- [Improvement] Add "local-docker-compose-lms/cms-dependencies" plugin
patches
- [Improvement] Use "exec" instead of "run" to initialise local platform
This has an impact on plugin hooks. Plugin hooks that needed to run
inside mysql-client now need to run inside mysql container. This
simplifies the deployment, as we no longer have an empty mysql-client
container sitting around.
When mysql is not enabled (ACTIVATE_MYSQL=False) the mysql container is
simply a mysql client.
We ran into an issue when trying to run migrations when the MinIO plugin
is activated. As seen in issues #243 and #244, the
certificates.0003_data__default_modes migration requires access to
MinIO. To do so, the MinIO host must be reached. That means that SSL
certificates must be in place (if https is enabled) and that the nginx
server must be booted. However, it does not make sense to require that
the minio container depends on the nginx container. So, in effect, we
need a fully working platform to run migrations.
In a sense, this is better as it harmonises the init task with k8s: in
k8s, init was already run with exec.
Next step is to get rid of these ugly mysql-client/minio-client
containers that must be up at all times. It would be much simpler to
just exec the commands inside the mysql/minio containers.
- [Feature] Modify ``createuser`` commands to define a password from the
command line
- [Improvement] Better yaml value parsing from command line
- [Feature] Add `dev exec` command
- [Bugfix] Fix incorrect notes settings definition
- [Improvement] Make it possible to start/stop/reboot a selection of
services
- [Improvement] Add `local/k8s reboot` commands
- [Improvement] Add `-U/--unset` option to `config save`
- [Bugfix] Fix insecure static asset loading when web proxy is enabled
- [Improvement] Rename `SECRET_KEY` configuration parameter to
`OPENEDX_SECRET_KEY`
- [Improvement] Add support for SSL and TLS in external SMTP server
(#231)
- [Bugfix] Fix missing video transcripts in LMS (#229)
- [Improvement] Make it possible to enable/disable multiple plugins at
once
- [Improvement] Add a few local and k8s patches for plugins
We have a side effect whenever we add a plugin manually to the tutor
configuration: the renderer does not update itself. We fix this by
checking the config at load time.
We achieve this thanks to a patch to edx-oauth2-provider:
https://github.com/edx/edx-oauth2-provider/pull/59
The network alias was required to circumvent url validation by the
oauth2 client creation tool. But the network alias was not compatible
with k8s deployment.
The port number is not included in the hostname when checking the list
of allowed values.
We also add the "lms"/"cms" values for interaction with the API from
within the internal docker network.
Video transcripts uploaded in the CMS were not visible in the LMS. This
was a symptom caused by the fact that the LMS and the CMS do not share
the same MEDIA_ROOT. We initially thought that data uploaded in the CMS
(such as transcripts) was stored in a shared data service, such as
mongodb. It is, in fact, not. This makes it even more important to run
an object storage service like minio for distributed services.
Close #229
The 0003 migration from the certificates app of the LMS requires that
the S3-like platform is correctly setup during initialisation. To solve
this issue, we introduce a pre-init hook that is run prior to the LMS
migrations.
P3P is obsolete https://en.wikipedia.org/wiki/P3P
Also, these P3P headers leak information about the application that is
running, making it sensitive to attacks.
- [Feature] Creation of a plugin system
- [Feature] Kubernetes support out of beta
- [Improvement] Switch to pinned image tags for easier upgrades
- 💥[Improvement] Remove the `-y/--yes` option: `tutor config save` is
now non-interactive by default. Use `-i/--interactive` to force
interactive mode.
- 💥[Improvement] Replace the `databases` command by `init`.
- [Improvement] Upgrade to ironwood.2
- [Improvement] Add `-y/--yes` option to `local quickstart` for
non-interactive quickstart
- [Improvement] Persist LMS/CMS logs to disk by default (with
collaboration from @silviot 💪)
- [Bugfix] Fix installing a locally cloned requirement repository
- [Improvement] Add `--no-cache` option to `images build`
- [Improvement] Make it possible to configure the notes service hostname
- [Improvement] Better, more robust MySQL initialisation
Having an identical "ironwood" tag for all releases is not practical, in
particular for breaking changes. Thus, docker images are now pinned to
the tutor version that they were build with.
Thus, we remove the -y/--yes options, which were kind of unintuitive,
and we add instead `-i/--interactive`. The quickstart commands remain
interactive by default, but can be silenced with `-I/--non-interactive`.
This commit introduces many changes:
- a fully functional minio plugin for local installation
- an almost-functional native k8s deployment
- a new way to process configuration, better suited to plugins
There are still many things to do:
- get rid of all the TODOs
- get a fully functional minio plugin for k8s
- add documentation for pluginso
- ...
Missing features:
- https certificates
- xqueue
- lms/cms workers
Moreover, we scalability issues due to the uploaded file storage in the
lms/cms. To address this issue we need to develop the MinIO plugin so
that it becomes compatible with Open edX.
Close #126#179#187
- [Improvement] Upgrade to ironwood.2
- [Improvement] Add `-y/--yes` option to `local quickstart` for non-interactive quickstart
- [Improvement] Persist LMS/CMS logs to disk by default (with collaboration from @silviot 💪)
- [Bugfix] Fix installing a locally cloned requirement repository
- [Improvement] Add `--no-cache` option to `images build`
- [Improvement] Make it possible to configure the notes service hostname
- [Improvement] Better, more robust MySQL initialization
This is especially useful on Kubernetes. With this change, the forum
container no longer crashes whenever mongodb or elasticsearch are
unavailable. Instead, it just waits for thoses services to be up.
Previously, we could not run forum migrations in Kubernetes because they
relied on exec-ing the migration command in the running container -- and
there was no such container, because the services where not already up.
Configuration loading was overly complex. Here, we simplify it
drastically with reasonable defaults. Hacky additional variables are
unncessary now that we use custom jinja2 filters.
For initializing the mysql database, we now run the mysql client from
the mysql container. That means that the mysql client is no longer
required in the openedx container, as previously suggested in PR #209.
When running `pip install -r /path/to/x.txt`, relative paths listed in
x.txt are supposed to be relative to the current folder, and not the
x.txt folder. This results in an error during `pip install`:
./xblock-poll/ should either be a path to a local project or a VCS
url beginning with svn+, git+, hg+, or bzr+
This error was first reported here:
https://discuss.overhang.io/t/installing-custom-xblock/61
Tutor is not using any bash-specific feature.
This commit changes (almost) all invocations to /bin/bash replacing them with
/bin/sh. It allows building a docker image that does not bundle bash in it.
The file k8s.py still has a bash invocation, to make sure this change does not
impact functionality for any user.
Given that each gunicorn worker uses ~200-250Mb virtual memory, we should
really avoid running too many of them. Here, we reduce the default
number of workers to 2. For the LMS and the CMS, this number can be
adjusted by setting an environment variable GUNICORN_WORKERS.
This is related to #207.
Settings the number of workers for the LMS and the CMS to 2*|CPU|+1
results in 4*|CPU]+2 gunicorn workers, which is too much for a laptop to
cope with.
Close #207
Migrations were being done on openedx-assets, which does not depend on
mysql. As a consequence, the mysql container was not being created on
"migrate" and migrations were failing with "Timeout after 20s".
Installing from pip was broken because, for some reason, the
templates/apps/xqueue/settings/ folder contained a __pycache__ folder
with compiled *.pyc files. Rendering the binary files was failing
miserably.
Login from localhost/studio.localhost was broken on Ironwood because the
session cookie was configured to be stored under the production domain
name. We fix this by setting the SESSION_COOKIE_DOMAIN to None: in the
edx-platform code, this corresponds to using the current request domain.
Now that the correct webpack settings are loaded by the `update_assets`
command in Ironwood, we can stop relying on the `openedx-assets` script.
Actually, we could probably remove it.
We were getting the following warning with the latest version of pyyaml:
YAMLLoadWarning: calling yaml.load() without Loader=... is
deprecated, as the default Loader is unsafe. Please read
https://msg.pyyaml.org/load for full details.
With the upgrade to Ironwood, studio login redirects to the LMS by
default. This breaks login from studio.localhost. So we activate SSO
login only when HTTPS is activated, which should happen only when the
domains are correctly configured.
- [Feature] Multiple platforms on a single server \o/
- [Feature] Easily configure web proxy on the host
- [Bugfix] Fix `images pull all` command which failed on "all" image
- [Improvement] Add configurable mongodb, SMTP and rabbitmq
authentication
- [Improvement] Harmonize mysql username/password configuration
parameters
- [Feature] Configurable and pluggable data storage backends (#114)
In the future, we want to allow users to rely on third-party services
for data storage, such as hosted MySQL and such. To do so, we need to be
able to configure the host/port of these services, which we do here.
This is to address part of #114.
It was tricky to define correct settings: the "localfs" default storage
backend was causing incorrect urls of the form "/media/<id>/<name>.csv".
Close #143
Configuration values can be loaded from the system environment by adding
a "TUTOR_" prefix.
Environment values supersede values from the user configuration file, so
that we can set values from the command line with "KEY=VAL tutor config
save --silent" even when KEY is already present in the user
configuration file.
Environment is no longer generated separately for each target, but only
once the configuration is saved.
Note that the environment is automatically updated during
re-configuration, based on a "version" file stored in the environment.
- [Improvement] Install python requirements in virtual env in docker
image
- [Bugfix] Add missing volume for theme development
- [Improvement] Rename "config [non]interactive" command to "config save
[--silent]"
- [Improvement] More explicit logging during environment generation
- [Improvement] Configurable docker images (#122)
- [Bugfix] Fix "android pullimage" command
- [Improvement] Do not upgrade images as part of quickstart
- [Bugfix] Fix USERID setup in development mode and various dev-related
docs (#177)
Pulling images is slow, often unnecessary, and sometimes suprising for
the user, as the newer images may break the older tutor version.
Instead, we add a "--pullimages" option to "quickstart" to explicitely
update docker images.
USERID environment variable was no longer passed to docker image in
development mode.
We take the opportunity to improve the documentation regarding the dev
environment.
Close #177.
We had to backtrack from the latest release of the
android app, which is not compatible with the mobile api v0.5 available
in Hawthorn. This should change in ironwood.
Also, we included the correct oauth client ID in the app, which
prevented communication with the LMS.
The android app is now out of beta \o/
Close #89
- [Minor] Minimum required `click` version is 7.0 (#171)
- [Bugfix] Fix `runserver` dev command (#172)
- [Minor] Fix non-https link to documentation in pypi
- [Minor] Fix `createuser` documentation
Mysql triggers "Host 'xxx' is not allowed to connect to this MySQL
server" error because of race condition.
This was an issue that had been (painfully) solved in v2 and has reared
its ugly head again in v3.
Close #159
Close #160
`./dist/tutor --version` raises an error:
Traceback (most recent call last):
File "main", line 4, in <module>
File "tutor/cli.py", line 21, in main
File "click/core.py", line 764, in __call__
File "click/core.py", line 716, in main
File "click/core.py", line 641, in make_context
File "click/core.py", line 1089, in parse_args
File "click/core.py", line 940, in parse_args
File "click/core.py", line 1477, in handle_parse_result
File "click/core.py", line 96, in invoke_param_callback
File "click/decorators.py", line 270, in callback
RuntimeError: Could not determine version
[1659] Failed to execute script main
To address this, we pass the version value directly to click.
This is for issue #156.
Replace all make commands by a single "tutor" binary. Environment and
data are all moved to ~/.tutor/local/share/tutor. We take the
opportunity to add a web UI and revamp the documentation.
This is a complete rewrite.
Close #121.
Close #147.
