- 💥 [Feature] Upgrade to Nutmeg: (by @regisb)
- 💥 [Feature] Persistent grades are now enabled by default.
- [Bugfix] Remove edX references from bulk emails ([issue](https://github.com/openedx/build-test-release-wg/issues/100)).
- [Improvement] For Tutor Nightly (and only Nightly), official plugins are now installed from their nightly branches on GitHub instead of a version range on PyPI. This will allow Nightly users to install all official plugins by running ``pip install -e ".[full]"``.
- [Bugfix] Start MongoDB when running migrations, because a new data migration fails if MongoDB is not running
Celery workers failed to start in development with the following stacktrace:
cms-worker_1 | Traceback (most recent call last):
cms-worker_1 | File "/openedx/venv/bin/celery", line 8, in <module>
cms-worker_1 | sys.exit(main())
cms-worker_1 | File "/openedx/venv/lib/python3.8/site-packages/celery/__main__.py", line 16, in main
cms-worker_1 | _main()
cms-worker_1 | File "/openedx/venv/lib/python3.8/site-packages/celery/bin/celery.py", line 322, in main
cms-worker_1 | cmd.execute_from_commandline(argv)
cms-worker_1 | File "/openedx/venv/lib/python3.8/site-packages/celery/bin/celery.py", line 499, in execute_from_commandline
cms-worker_1 | super(CeleryCommand, self).execute_from_commandline(argv)))
cms-worker_1 | File "/openedx/venv/lib/python3.8/site-packages/celery/bin/base.py", line 305, in execute_from_commandline
cms-worker_1 | return self.handle_argv(self.prog_name, argv[1:])
cms-worker_1 | File "/openedx/venv/lib/python3.8/site-packages/celery/bin/celery.py", line 491, in handle_argv
cms-worker_1 | return self.execute(command, argv)
cms-worker_1 | File "/openedx/venv/lib/python3.8/site-packages/celery/bin/celery.py", line 415, in execute
cms-worker_1 | return cls(
cms-worker_1 | File "/openedx/venv/lib/python3.8/site-packages/celery/bin/worker.py", line 221, in run_from_argv
cms-worker_1 | *self.parse_options(prog_name, argv, command))
cms-worker_1 | File "/openedx/venv/lib/python3.8/site-packages/celery/bin/base.py", line 428, in parse_options
cms-worker_1 | self.parser = self.create_parser(prog_name, command)
cms-worker_1 | File "/openedx/venv/lib/python3.8/site-packages/celery/bin/base.py", line 440, in create_parser
cms-worker_1 | description=self._format_description(self.description),
cms-worker_1 | File "/openedx/venv/lib/python3.8/site-packages/celery/bin/base.py", line 462, in _format_description
cms-worker_1 | text.fill_paragraphs(text.dedent(description), width))
cms-worker_1 | File "/openedx/venv/lib/python3.8/site-packages/celery/utils/text.py", line 58, in fill_paragraphs
cms-worker_1 | return sep.join(fill(p, width) for p in s.split(sep))
cms-worker_1 | File "/openedx/venv/lib/python3.8/site-packages/celery/utils/text.py", line 58, in <genexpr>
cms-worker_1 | return sep.join(fill(p, width) for p in s.split(sep))
cms-worker_1 | File "/opt/pyenv/versions/3.8.12/lib/python3.8/textwrap.py", line 391, in fill
cms-worker_1 | return w.fill(text)
cms-worker_1 | File "/opt/pyenv/versions/3.8.12/lib/python3.8/textwrap.py", line 363, in fill
cms-worker_1 | return "\n".join(self.wrap(text))
cms-worker_1 | File "/opt/pyenv/versions/3.8.12/lib/python3.8/textwrap.py", line 354, in wrap
cms-worker_1 | return self._wrap_chunks(chunks)
cms-worker_1 | File "/opt/pyenv/versions/3.8.12/lib/python3.8/textwrap.py", line 248, in _wrap_chunks
cms-worker_1 | raise ValueError("invalid width %r (must be > 0)" % self.width)
cms-worker_1 | ValueError: invalid width -2 (must be > 0)
This issue was reported upstream here: https://github.com/celery/celery/issues/6302
It is caused by the `tty: true` statement, for some reason. It will be fixed in
Nutmeg, after celery is upgraded to 5.2.6.
Close #681.
- [Security] Apply logout redirect url security fix. (by @regisb)
- [Feature] Make it possible to force the rendering of a given template, even when the template path matches an ignore pattern. (by @regisb)
- 💥[Fix] Get rid of the `tutor config render` command, which is useless now that themes can be implemented as plugins. (by @regisb)
- [Security] Apply logout redirect url security fix. (by @regisb)
- [Feature] Make it possible to force the rendering of a given template, even when the template path matches an ignore pattern. (by @regisb)
- 💥[Fix] Get rid of the `tutor config render` command, which is useless now that themes can be implemented as plugins. (by @regisb)
When rendering theme files in a plugin, the *.scss files are stored in a
"partials" subdirectory, which was ignored by the environment rendering logic.
To render these files, we move the path ignoring logic to a filter, which is a
list of regular expressions. Values in this filter can be overridden by another
filter.
See the corresponding issue in the indigo theme plugin:
https://github.com/overhangio/tutor-indigo/issues/24
- [Fix] Truncate site display name to 50 characters with a warning, fixing data too long error for long site names. (by @navinkarkera)
- [Feature] Add patch to allow overriding final openedx docker image CMD.
- [Fix] Ignore Python plugins that cannot be loaded. (by @regisb)
- [Improvement] Faster and more reliable builds with `npm clean-install` instead of `npm install`. (by @regisb. Thanks @ghassanmas!)
- [Fix] Fix 500 error during studio login. (by @regisb)
- [Fix] Fix updates for the Caddy deployment in multi-node Kubernetes clusters (#660). Previously, Caddy configuration updates might fail if the Kubernetes cluster had more than one worker node. (by @fghaas)
When running multiple concurrent versions of a plugin there are sometimes
version conflicts that prevent the plugin from being loaded. Prior to v1, Tutor
was correctly ignoring plugins that could not be loaded. During the transition
to v1 we lost that feature because we only captured TutorErrors.
When a Pod associated with a Deployment is updated (for example, due
to a change to its ConfigMap, or an updated image reference),
Kubernetes uses a ReplicaSet to spin up a Pod with the new
configuration, and once it is up, it tears down the old one.
In case of the Caddy Deployment, this is complicated by the fact that
it uses a Persistent Volume Claim (PVC), whose corresponding volume
uses a Read/Write-Once (RWO) configuration. This means that it can
only be used by multiple Pods if all those Pods all run on the same
Kubernetes worker node.
In order to enable rolling upgrades for the Caddy Deployment, we need
to ensure that its replacement Pod is scheduled on the same node as
the original Pod.
Thus, add a pod affinity rule that will force exactly that behavior.
Reference:
https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/
The other Tutor services that use volumes (MySQL, Redis, Elasticsearch
and MongoDB) do not need this fix, since they all use the "Recreate"
deployment strategy: their Pods are all automatically torn down before
being replaced. This strategy is not needed for Caddy, and using a pod
affinity rule is less disruptive to the learner experience.
Commit 514e3fce22 made it so
that dev containers were to load mounts from
env/dev/docker-compose.tmp.yml. However, it did not update
the code to generates the docker-compose.tmp.yml files.
This manifested as mounts simply not working in dev mode.
Additionally, we make the docker-compose.jobs.tmp.yml files
follow the same local vs dev differentiation that
was introduced in 514e3fce22.
When mounting a directory in a dev-only container, such as the
"learning" mfe, docker-compose is failing because it is attempting to
run "docker-compose stop" in the local context -- which knows nothing
about the learning container.
To resolve this, we store tmp volumes either in the local or dev
docker-compose.yml, and load either one depending on the context.
- [Improvement] Add the `COMPOSE_PROJECT_STARTED` action and run `dev
stop` on `local start` (and vice versa).
- [Feature] Introduce `local/dev copyfrom` command to copy contents from
a container.
- [Bugfix] Fix a race condition that could prevent a newly provisioned
LMS container from starting due to a `FileExistsError` when creating
data folders.
- [Deprecation] Mark `tutor dev runserver` as deprecated in favor of
`tutor dev start`. Since `start` now supports bind-mounting and
breakpoint debugging, `runserver` is redundant and will be removed in a
future release.
- [Improvement] Allow breakpoint debugging when attached to a service
via `tutor dev start SERVICE`.
- [Security] Apply rate limiting security fix (see
[commit](b5723e416e)).
- [Feature] Introduce the ``-m/--mount`` option in ``local`` and ``dev``
commands to auto-magically bind-mount folders from the host.
- [Feature] Add `tutor dev quickstart` command, which is similar to
`tutor local quickstart`, except that it uses dev containers instead
of local production ones and includes some other small differences for
the convience of Open edX developers. This should remove some friction
from the Open edX development setup process, which previously required
that users provision using local producation containers (`tutor local
quickstart`) but then stop them and switch to dev containers (`tutor
local stop && tutor dev start -d`).
- 💥[Improvement] Make it possible to run `tutor k8s exec <command with
multiple arguments>` (#636). As a consequence, it is no longer
possible to run quoted commands: `tutor k8s exec "<some command>"`.
Instead, you should remove the quotes: `tutor k8s exec <some command>`.
- 💥[Deprecation] Drop support for the `TUTOR_EDX_PLATFORM_SETTINGS`
environment variable. It is now recommended to create a plugin
instead.
- 💥[Improvement] Complete overhaul of the plugin extension mechanism.
Tutor now has a hook-based Python API: actions can be triggered at
different points of the application life cycle and data can be modified
thanks to custom filters. The v0 plugin API is still supported, for
backward compatibility, but plugin developers are encouraged to migrate
their plugins to the new API. See the new plugin tutorial for more
information.
- [Improvement] Improved the output of `tutor plugins list`.
- [Feature] Add `tutor [dev|local|k8s] status` command, which provides
basic information about the platform's status.
- [Improvement] Add the `COMPOSE_PROJECT_STARTED` action and run `dev
stop` on `local start` (and vice versa).
- [Feature] Introduce `local/dev copyfrom` command to copy contents from
a container.
- [Bugfix] Fix a race condition that could prevent a newly provisioned
LMS container from starting due to a `FileExistsError` when creating
data folders.
- [Deprecation] Mark `tutor dev runserver` as deprecated in favor of
`tutor dev start`. Since `start` now supports bind-mounting and
breakpoint debugging, `runserver` is redundant and will be removed in a
future release.
- [Improvement] Allow breakpoint debugging when attached to a service
via `tutor dev start SERVICE`.
- [Security] Apply rate limiting security fix (see
[commit](b5723e416e)).
- [Feature] Introduce the ``-m/--mount`` option in ``local`` and ``dev``
commands to auto-magically bind-mount folders from the host.
- [Feature] Add `tutor dev quickstart` command, which is similar to
`tutor local quickstart`, except that it uses dev containers instead
of local production ones and includes some other small differences for
the convience of Open edX developers. This should remove some friction
from the Open edX development setup process, which previously required
that users provision using local producation containers (`tutor local
quickstart`) but then stop them and switch to dev containers (`tutor
local stop && tutor dev start -d`).
- 💥[Improvement] Make it possible to run `tutor k8s exec <command with
multiple arguments>` (#636). As a consequence, it is no longer
possible to run quoted commands: `tutor k8s exec "<some command>"`.
Instead, you should remove the quotes: `tutor k8s exec <some command>`.
- 💥[Deprecation] Drop support for the `TUTOR_EDX_PLATFORM_SETTINGS`
environment variable. It is now recommended to create a plugin
instead.
- 💥[Improvement] Complete overhaul of the plugin extension mechanism.
Tutor now has a hook-based Python API: actions can be triggered at
different points of the application life cycle and data can be modified
thanks to custom filters. The v0 plugin API is still supported, for
backward compatibility, but plugin developers are encouraged to migrate
their plugins to the new API. See the new plugin tutorial for more
information.
- [Improvement] Improved the output of `tutor plugins list`.
- [Feature] Add `tutor [dev|local|k8s] status` command, which provides
basic information about the platform's status.
Running `local start` while a dev platform is still running is a common sourse
of mistakes. Here we introduce a new action to automatically stop local and dev
projects whenever a project with a different name is started.
`copyfrom` copies data from a container to the local filesystem. It's similar
to bindmount, but less clunky, and more intuitive. Also, it plays along great
with `--mount`. Eventually we'll just get rid of the `bindmount` command and
the `--volume` option.
Previously, `tutor dev runserver --volume=x --volume=y`
would log:
'runserver' is deprecated and will be removed in a future release.
Use 'start' instead. Bind-mounts can be specified using '-m/--mount'.
Bind-mounts can be specified using '-m/--mount'.
`tutor dev runserver` will be removed in a future release.
Developers are encouraged to use `tutor dev start` instead,
which is more flexible and provides a consistent interface
with `tutor local start`.
As part of this deprecation, we enable the `tty` and
`stdin_open` options on development docker-compose
services. This will allow developers to use `start`
for breakpoint debugging, which was previously only
availble via `runserver`. Several parallel PRs have
been merged in order to make the same change in the
development services of the official plugins.
Although `start` does not support the `--volume` option,
it supports a more-powerful `--mount` option. So, where
developers previously used:
tutor dev runserver --volume ...
to bind-mount host directories, they should now use:
tutor dev start --mount ...
Resolves https://github.com/overhangio/2u-tutor-adoption/issues/61
When a v1 plugin was installed, several things were happening regarding tests:
1. v1 plugin loading was happening despite the TUTOR_IGNORE_ENTRYPOINT_PLUGINS
environment variable.
2. the CORE_READY event was not triggered because it was happening just once at
import time.
This was causing some tests to incorrectly load the MFE plugin.
The `--mount` option is available both with `tutor local`
and `tutor dev` commands. It allows users to easily bind-mount containers from
the host to containers. Yes, I know, we already provide that possibility with
the `bindmount` command and the `--volume=/path/` option. But these suffer from
the following drawbacks:
- They are difficult to understand.
- The "bindmount" command name does not make much sense.
- It's not convenient to mount an arbitrary folder from the host to multiple
containers, such as the many lms/cms containers (web apps, celery workers and
job runners).
To address this situation, we now recommend to make use of --mount:
1. `--mount=service1[,service2,...]:/host/path:/container/path`: manually mount
`/host/path` to `/container/path` in container "service1" (and "service2").
2. `--mount=/host/path`: use the new v1 plugin API to discover plugins that
will detect this option and select the right containers in which to bind-mount
volumes. This is really nifty...
Close https://github.com/overhangio/2u-tutor-adoption/issues/43
Add `tutor dev quickstart` command, which is equivalent to
`tutor local quickstart`, but uses dev containers instead
of local production ones and includes some other small
differences for the convience of Open edX developers.
This should remove some friction
from the Open edX development setup process, which previously
required that users provision using local producation
containers but then stop them and switch to dev containers:
* tutor local quickstart
* tutor local stop
* tutor dev start -d
Document the command and its improved workflow in
./docs/tutorials/nightly.rst
Fixes overhangio/2u-tutor-adoption#58
The entrypoint in the "openedx" Docker image was used only to define the
DJANGO_SETTINGS_MODULE environment variable, based on SERVICE_VARIANT and
SETTINGS. We ditch SETTINGS in favour of defining explicitely
DJANGO_SETTINGS_MODULE.
The problem with the Docker entrypoint is that it was bypassed whenever we ran
`tutor local exec` or `tutor k8s exec`. By removing it we make it simpler for
end-users to run manage.py commands in kubernetes.
Previously, it was possible to override settings by defining the
TUTOR_EDX_PLATFORM_SETTINGS environment variable. But let's face it:
- It was not very well supported.
- It was poorly explained.
- It was not very useful.
- It causes unnecessary code complexity.
For these reasons, we drop that feature.
This is a very large refactoring which aims at making Tutor both more
extendable and more generic. Historically, the Tutor plugin system was
designed as an ad-hoc solution to allow developers to modify their own
Open edX platforms without having to fork Tutor. The plugin API was
simple, but limited, because of its ad-hoc nature. As a consequence,
there were many things that plugin developers could not do, such as
extending different parts of the CLI or adding custom template filters.
Here, we refactor the whole codebase to make use of a generic plugin
system. This system was inspired by the Wordpress plugin API and the
Open edX "hooks and filters" API. The various components are added to a
small core thanks to a set of actions and filters. Actions are callback
functions that can be triggered at different points of the application
lifecycle. Filters are functions that modify some data. Both actions and
filters are collectively named as "hooks". Hooks can optionally be
created within a certain context, which makes it easier to keep track of
which application created which callback.
This new hooks system allows us to provide a Python API that developers
can use to extend their applications. The API reference is added to the
documentation, along with a new plugin development tutorial.
The plugin v0 API remains supported for backward compatibility of
existing plugins.
Done:
- Do not load commands from plugins which are not enabled.
- Load enabled plugins once on start.
- Implement contexts for actions and filters, which allow us to keep track of
the source of every hook.
- Migrate patches
- Migrate commands
- Migrate plugin detection
- Migrate templates_root
- Migrate config
- Migrate template environment globals and filters
- Migrate hooks to tasks
- Generate hook documentation
- Generate patch reference documentation
- Add the concept of action priority
Close #499.
MySQL 8 drop the support for creating users by executing `GRANT ALL`. This commit splits the user creation and permission granting, therefore the newer MySQL versions are supported too.
MySQL 8 is supported by edx-platform: 1cdb0347c5/playbooks/roles/mysql/tasks/mysql.yml (L93-L98)
PR #619 set the EDX_PLATFORM_VERSION build arg's default to
OPENEDX_COMMON_VERSION. While this works fine for setting a
non-default branch to run edx code from (say, "master"), it may break
if the user sets OPENEDX_COMMON_VERSION to a branch or tag name that
does not exist upstream in repositories *other than*
EDX_PLATFORM_REPOSITORY.
Thus, introduce a separate configuration parameter,
EDX_PLATFORM_VERSION, to match the build arg of the same name. Set its
default to OPENEDX_COMMON_VERSION.
This way, the user can deploy an arbitrarily-named fork of
edx-platform, while retaining the default OPENEDX_COMMON_VERSION
(like, for example "open-release/maple.3") for everything else.
Previously, the `k8s exec` command did not support unknown "--options". This
made it impossible to launch, say, a django shell in the lms container.
While implementing this feature we saw an opportunity to simplify the way jobs
are handled in the k8s commands.
Close #636.
Another related issue is: https://github.com/overhangio/2u-tutor-adoption/issues/52
- [Security] Apply SAML security fix.
- [Improvement] In addition to the Docker build arguments
`EDX_PLATFORM_REPOSITORY` and `NPM_REGISTRY`, also support two corresponding
and identically-named `config.yml` values serving the same purpose.
Previously, the only way for Tutor users to use a fork of edx-platform
or a custom NPM registry was to use build args during the image build.
This is suboptimal in the case of automatically building images from
CI pipelines, which may want to auto-detect when an image needs to be
rebuilt based on config.yml changes.
In addition, the EDX_PLATFORM_VERSION build argument can already be
set via a corresponding config.yml parameter (OPENEDX_COMMON_VERSION),
so it's reasonable to follow that precedent and also introduce
config.yml parameters to correspond with the EDX_PLATFORM_REPOSITORY
and NPM_REGISTRY build arguments.
Thus, introduce two new configuration parameters:
- EDX_PLATFORM_REPOSITORY
- NPM_REGISTRY
These parameters can now optionally be used instead of the
aforementioned build args.
Before edx-platform was upgraded to Celery 5, lms-worker and
cms-worker could be invoked using this syntax:
celery worker --app=APP <args> --maxtasksperchild=N <args>
Since the recent Celery 5 upgrade (edx-platform commit 0588c92),
though, this fails with the messages:
You are using `--app` as an option of the worker sub-command:
celery worker --app celeryapp <...>
The support for this usage was removed in Celery 5.0.
Instead you should use `--app` as a global option:
celery --app celeryapp worker <...>
and:
Error: No such option: --maxtasksperchild
(Possible options: --max-memory-per-child, --max-tasks-per-child)
So, this commit changes the lms-worker and cms-worker invocations to:
celery --app=APP <args> --max-tasks-per-child=N <args>
- [Bugfix] Fix dockerize on arm64 by switching to the [powerman/dockerize](https://github.com/powerman/dockerize) fork (#591).
- [Bugfix] Fix "Unexpected args" error during service initialization on Kubernetes (#611).
The version of dockerize that shipped with the "openedx" image was not
compatible with arm64. The original project is unmaintained, but there
is a fork that provides a version that is compatible with arm64.
This was tested on arm64 with buildx:
docker buildx build --tag=openedx --platform=linux/arm64 ~/.local/share/tutor/env/build/openedx
Close #591
- [Bugfix] Fix `local/k8s quickstart` commands when upgrading from an older release (#595).
- [Bugfix] Fix running the default exim-relay SMTP server on arm64 (#600).
- [Feature] Add `tutor k8s apply` comand, which is a direct interface with `kubectl apply`.
- [Feature] Add `openedx-dockerfile-minimal` patch, which you can use to install custom packages and run commands as root in the Docker image.
- [Security] Fix vulnerability in call to invalid enrollment API (see [commit](e9369cffde)).
- [Bugfix] Fix "Internal Server Error / AttributeError / object has no attribute 'get_metadata'" in learning MFE.
- [Improvement] Replace all links to github.com/edx by github.com/openedx, following the migration of all repositories.
- [Bugfix] Fix `k8s start caddy` command.
- [Bugfix] Fix authentication in development due to missing SameSite policy on session ID cookie.
- [Bugfix] Display properly themed favicon.ico image in LMS, Studio and microfrontends.
- [Bugfix] Fix "LazyStaticAbsoluteUrl is not JSON serializable" error when sending bulk emails.
- [Bugfix] Fix `tutor local importdemocourse` fails when platform is not up.
In development, it was no longer possible to authenticate to the lms. Ater
signing in, the session ID could not be dropped, and thus the user was not
signed in, although no error was logged -- just a warning in the browser
console.
This problem was caused by the fact that the SameSite policy was set to "None"
in development.
Previously, we were redirecting all /*favicon.ico requests to the default
favicon. This meant that the favicon might not necessarily be correctly themed,
most notably in MFEs. Here, we resolve this issue by redirecting to the
theme-agnostic theming/asset/* url. Also, we restrict the overly generic regexp
for favicon url matching. We verified that we did not miss any url by running
the following command on the demo server:
tutor local logs caddy | grep --only-matching "host.*favicon.ico" | sort | uniq
The LazyStaticAbsoluteUrl object was breaking bulk emails again with the
following stacktrace:
2022-01-11 13:50:10,591 ERROR 12 [celery.app.trace] [user None] [ip None] trace.py:255 - Task lms.djangoapps.instructor_task.tasks.send_bulk_course_email[26b93357-018a-408f-b3f7-b69722447c5b] raised unexpected: EncodeError(TypeError('Object of type LazyStaticAbsoluteUrl is not JSON serializable'))
Traceback (most recent call last):
File "/openedx/venv/lib/python3.8/site-packages/kombu/serialization.py", line 50, in _reraise_errors
yield
File "/openedx/venv/lib/python3.8/site-packages/kombu/serialization.py", line 221, in dumps
payload = encoder(data)
File "/openedx/venv/lib/python3.8/site-packages/kombu/utils/json.py", line 69, in dumps
return _dumps(s, cls=cls or _default_encoder,
File "/openedx/venv/lib/python3.8/site-packages/simplejson/__init__.py", line 398, in dumps
return cls(
File "/openedx/venv/lib/python3.8/site-packages/simplejson/encoder.py", line 296, in encode
chunks = self.iterencode(o, _one_shot=True)
File "/openedx/venv/lib/python3.8/site-packages/simplejson/encoder.py", line 378, in iterencode
return _iterencode(o, 0)
File "/openedx/venv/lib/python3.8/site-packages/kombu/utils/json.py", line 59, in default
return super(JSONEncoder, self).default(o)
File "/openedx/venv/lib/python3.8/site-packages/simplejson/encoder.py", line 272, in default
raise TypeError('Object of type %s is not JSON serializable' %
TypeError: Object of type LazyStaticAbsoluteUrl is not JSON serializable
The point of that lazy object was to link to the lms logo even when a custom
theme was enabled. Luckily, we no longer need this lazy evaluation because we
now have theme-agnostic urls that point to static asset (see
https://github.com/openedx/edx-platform/pull/29461).
See:
https://discuss.overhang.io/t/error-while-sending-bulk-emails-lazystaticabsoluteurl-is-not-json-serializable/2176/
In the LMS/CMS Dockerfile, the env var STUDIO_CFG is set
in order to point CMS at its configuration json/yaml file.
Since https://github.com/edx/edx-platform/pull/29534
(which introduced 0013-cms-vs-studio.rst), the STUDIO_CFG
variable has been deprecated in favor of CMS_CFG.
This change updates the Dockerfile to reflect the new
preferred environment variable.
The only noticeable impact of this change is that it
will remove a depreation warning from Django startup
for tutor uses running off of Open edX master.
upgrading.
- [Bugfix] During upgrade, make sure that environment is up-to-date
prior to prompting to rebuild the custom images.
- [Bugfix] Fix ownership of mysql data, in particular when upgrading a
Kubernetes cluster to Maple.
- [Bugfix] Ensure that ``tutor k8s upgrade`` is run during ``tutor k8s
quickstart``, when necessary.
- 💥[Bugfix] By default, detect the current version during ``tutor
k8s/local upgrade``.
- [Bugfix] Fix upgrading from Lilac to Maple on Kubernetes by deleting
deployments and services.
`upgrade` had several issues, which are summarized here:
https://discuss.overhang.io/t/confusing-instructions-during-upgrade/2281/7
- The docs say that you should run quickstart, but what most people will see is
the big command tutor local upgrade --from=lilac verbatim paragraph.
- The local upgrade command should be very explicit about the fact that users
need to run quickstart.
- Maybe the name of the local upgrade command should be improved.
- When upgrading tutor from one major release to the next, there should be a
more explicit warning to inform users of what they are doing (see this other
conversation 1)
- We should tell people that they almost certainly need to enable the tutor and
the mfe plugins, if they are not enabled during upgrade.
- A link to all of the breaking changes from the changelog should be
prominently displayed during upgrade.
- The docs should emphasize that upgrading from one major release to the next
is potentially a risky endeavor and that downgrading is not possible. The docs
should also link to the changelog.
This commit has grown slightly beyond the intended scope, but the changes should be mostly positive.
In theory, we can assign ownership of mysql data to just any user. But in
Lilac, mysql was running with user 999. When upgrading to Maple, on Kubernetes,
the fsGroupChangePolicy was causing a change of the data *group* (to 1000) but
not of the user. This was causing a crash with the following error:
[ERROR] InnoDB: The error means mysqld does not have the access rights to the directory.
When upgrading from Lilac, all services break with the following error:
Service "***" is invalid: spec.ports[0].nodePort: Forbidden: may not be used when `type` is 'ClusterIP'
Upgrading deployments fails as well:
Deployment.apps "***" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{"app.kubernetes.io/instance":"openedx-********", "app.kubernetes.io/managed-by":"tutor", "app.kubernetes.io/name":"***", "app.kubernetes.io/part-of":"openedx"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable
That's because deployments and services need to be deleted as part of the Maple
upgrade. So that's what we do as part of `tutor k8s upgrade --from=lilac`. And
we take the opportunity to:
1. Run upgrade as part of quickstart, when necessary.
2. Default to lilac during `tutor k8s upgrade`.
Close #551.
- [Security] Upgrade Django to 3.2.11 in edx-platform.
- [Security] Prevent non-staff users from searching usernames by email by
abusing the logout url.
Without this patch, it is possible to search for account info including
username by using the email of a learner. This fix disallows searching using
email by regular users and restricts this feature to only staff and superusers.
- A shared cookie domain between lms and cms is no longer recommended:
https://github.com/edx/edx-platform/blob/master/docs/guides/studio_oauth.rst
- refactor: clean mounted data folder in lms/cms. In Lilac, the
bind-mounted lms/data and cms/data folders are a mess because new
folders are created there for every new course organisation. These
folders are empty. As far as we know they are useless... With this
change we move these folders to a dedicated "modulestore" subdirectory;
which corresponds better to the initial intent of the fs_root setting.
- fix: frontend failure during login to the lms. See:
https://github.com/openedx/build-test-release-wg/issues/104
- feat: move all forum-related code to a dedicated plugin. Forum is an
optional feature, and as such it deserves its own plugin. Starting from
Maple, users will be able to install the forum from
https://github.com/overhangio/tutor-forum/
- migrate from DCS_* session cookie settings to SESSION_*. That's
because edx-platform no longer depends on django-cookies-samesite. Close
https://github.com/openedx/build-test-release-wg/issues/110
- get rid of tons of deprecation warnings in the lms/cms
- feat: make it possible to point to themed assets. Cherry-picking this
change makes it possible to point to themed assets with a theme-agnostic
url, notably from MFEs.
- Install all official plugins as part of the `tutor[full]` package.
- Don't print error messages about loading plugins during autocompletion.
- Prompt for image building when upgrading from one release to the next.
- Add `tutor local start --skip-build` option to skip building Docker images.
Close #450.
Close #545.
This introduces quite a few changes to make it easier to run Caddy as a load
balancer in Kubernetes:
- Make it possible to start/stop a selection of resources with ``tutor k8s
start/stop [names...]``.
- Make it easy to deploy an independent LoadBalancer by converting the caddy
service to a NodePort when ``ENABLE_WEB_PROXY=false``.
- Add a ``app.kubernetes.io/component: loadbalancer`` label to the LoadBalancer
service.
- Add ``app.kubernetes.io/name`` labels to all services.
- Preserve the LoadBalancer service in ``tutor k8s stop`` commands.
- Wait for the caddy deployment to be ready before running initialisation jobs.
Close #532.
On some providers (notably: DigitalOcean) NodePort services are not exposed to
the outside world. But this is not what the Kubernetes spec describes:
https://kubernetes.io/docs/concepts/services-networking/service/#publishing-services-service-types
Thus, there is a risk that NodePort services are exposed to the outside world
in some context. To avoid this, we convert all NodePort to ClusterIP resources.
Previously, configuration management was very confusing because we kept mixing
"base" and "defaults" configuration:
- It was difficult to make the difference between core settings that were
necessary (e.g: passwords) as opposed to others that could simply be
defaulted to.
- The order of settings in config.yml mattered: config entries that depended on
other needed to be defined later. As a consequence, Tutor was not compatible
with Python 3.5, where dict entries are not sorted.
Through the commonLabels directive in kustomization.yml, all resources
get a label named "app.kubernetes.io/version", which is being set to
the Tutor version at the time of initial deployment.
When the user then subsequently progresses to a new Tutor version,
Kubernetes attempts to update this label — but for Deployment,
ReplicaSet, and DaemonSet resources, this is no longer allowed as of
https://github.com/kubernetes/kubernetes/issues/50808. This causes
"tutor k8s start" (at the "kubectl apply --kustomize" step) to break
with errors such as:
Deployment.apps "redis" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{"app.kubernetes.io/instance":"openedx-JIONBLbtByCGUYgHgr4tDWu1", "app.kubernetes.io/managed-by":"tutor", "app.kubernetes.io/name":"redis", "app.kubernetes.io/part-of":"openedx", "app.kubernetes.io/version":"12.1.7"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable
Simply removing the app.kubernetes.io/version label from
kustomization.yml will permanently fix this issue for newly created
Kubernetes deployments, which will "survive" any future Tutor version
changes thereafter.
However, *existing* production Open edX deployments will need to throw
the affected Deployments away, and re-create them.
Also, add the Tutor version as a resource annotation instead, using
the commonAnnotations directive.
See also:
https://github.com/kubernetes/client-go/issues/508https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/commonlabels/https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/commonannotations/
Fixes #531.
When nginx was removed in favour of caddy, we decided that plugin
implementations of the "caddyfile" patch should make use of the "port" local
variable. However, local variables are not available from inside plugin
patches, which are rendered outside of the context of the parent templates.
For a more extensive description of the problem, see:
https://github.com/overhangio/tutor-mfe/pull/23#issuecomment-964016190
We still want to make it easy for developers to decide what should the port be
for caddy hosts. To do so, we make use of environment variables that are passed
at runtime to the caddy container.
Thus, a regular plugin patch should look like this:
{{ PLUGIN_HOST }}{$default_site_port} {
import proxy "myplugin:8000"
}
Forum is an optional feature, and as such it deserves its own plugin. Starting
from Maple, users will be able to install the forum from
https://github.com/overhangio/tutor-forum/
Close #450.
With this change, containers are no longer run as "root" but as unprivileged
users. This is necessary in some environments, notably some Kubernetes
clusters.
To make this possible, we need to manually fix bind-mounted volumes in
docker-compose. This is pretty much equivalent to the behaviour in Kubernetes,
where permissions are fixed at runtime if the volume owner is incorrect. Thus,
we have a consistent behaviour between docker-compose and Kubernetes.
We achieve this by bind-mounting some repos inside "*-permissions" services.
These services run as root user on docker-compose and will fix the required
permissions, as per build/permissions/setowner.sh These services simply do not
run on Kubernetes, where we don't rely on bind-mounted volumes. There, we make
use of Kubernete's built-in volume ownership feature.
With this change, we get rid of the "openedx-dev" Docker image, in the sense
that it no longer has its own Dockerfile. Instead, the dev image is now simply
a different target in the multi-layer openedx Docker image. This makes it much
faster to build the openedx-dev image.
Because we declare the APP_USER_ID in the dev/docker-compose.yml file, we need
to pass the user ID from the host there. The only way to achieve that is with a
tutor config variable. The downside of this approach is that the
dev/docker-compose.yml file is no longer portable from one machine to the next.
We consider that this is not such a big issue, as it affects the development
environment only.
We take this opportunity to replace the base image of the "forum" image. There
is now no need to re-install ruby inside the image. The total image size is
only decreased by 10%, but re-building the image is faster.
In order to run the smtp service as non-root, we switch from namshi/smtp to
devture/exim-relay. This change should be backward-compatible.
Note that the nginx container remains privileged. We could switch to
nginxinc/nginx-unprivileged, but it's probably not worth the effort, as we are
considering to get rid of the nginx container altogether.
Close #323.
This introduces quite a few changes to make it easier to run Caddy as a load
balancer in Kubernetes:
- Make it possible to start/stop a selection of resources with ``tutor k8s
start/stop [names...]``.
- Make it easy to deploy an independent LoadBalancer by converting the caddy
service to a NodePort when ``ENABLE_WEB_PROXY=false``.
- Add a ``app.kubernetes.io/component: loadbalancer`` label to the LoadBalancer
service.
- Add ``app.kubernetes.io/name`` labels to all services.
- Preserve the LoadBalancer service in ``tutor k8s stop`` commands.
- Wait for the caddy deployment to be ready before running initialisation jobs.
Close #532.
On some providers (notably: DigitalOcean) NodePort services are not exposed to
the outside world. But this is not what the Kubernetes spec describes:
https://kubernetes.io/docs/concepts/services-networking/service/#publishing-services-service-types
Thus, there is a risk that NodePort services are exposed to the outside world
in some context. To avoid this, we convert all NodePort to ClusterIP resources.
- 💥[Improvement] Fail on incorrect image name argument in `images
build/pull/push/printtag` commands.
- [Bugfix] Remove trailing slashes in docker-compose files for
[compatibility with docker-compose v2 in
WSL](https://github.com/docker/compose/issues/8558).
- [Improvement] `settheme` now works with preview domain.
- [Feature] Allow specifying extra pip packages through config.yml.
Previously, configuration management was very confusing because we kept mixing
"base" and "defaults" configuration:
- It was difficult to make the difference between core settings that were
necessary (e.g: passwords) as opposed to others that could simply be
defaulted to.
- The order of settings in config.yml mattered: config entries that depended on
other needed to be defined later. As a consequence, Tutor was not compatible
with Python 3.5, where dict entries are not sorted.
Added OPENEDX_EXTRA_PIP_REQUIREMENTS setting, which allows to specify
extra pip packages that should be installed.
Moved "openedx-scorm-xblock" package from Dockerfile to the new setting
in the config.yml.
Through the commonLabels directive in kustomization.yml, all resources
get a label named "app.kubernetes.io/version", which is being set to
the Tutor version at the time of initial deployment.
When the user then subsequently progresses to a new Tutor version,
Kubernetes attempts to update this label — but for Deployment,
ReplicaSet, and DaemonSet resources, this is no longer allowed as of
https://github.com/kubernetes/kubernetes/issues/50808. This causes
"tutor k8s start" (at the "kubectl apply --kustomize" step) to break
with errors such as:
Deployment.apps "redis" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{"app.kubernetes.io/instance":"openedx-JIONBLbtByCGUYgHgr4tDWu1", "app.kubernetes.io/managed-by":"tutor", "app.kubernetes.io/name":"redis", "app.kubernetes.io/part-of":"openedx", "app.kubernetes.io/version":"12.1.7"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable
Simply removing the app.kubernetes.io/version label from
kustomization.yml will permanently fix this issue for newly created
Kubernetes deployments, which will "survive" any future Tutor version
changes thereafter.
However, *existing* production Open edX deployments will need to throw
the affected Deployments away, and re-create them.
Also, add the Tutor version as a resource annotation instead, using
the commonAnnotations directive.
See also:
https://github.com/kubernetes/client-go/issues/508https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/commonlabels/https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/commonannotations/
Fixes #531.
When running "tutor k8s" we can disable the forum deployment and
service by setting RUN_FORUM to false, but the corresponding job is
added to jobs.yml unconditionally.
Add a conditional to tutor/templates/k8s/jobs.yml so that the forum
job definition is only included when RUN_FORUM is true.
Fixes #525.
When nginx was removed in favour of caddy, we decided that plugin
implementations of the "caddyfile" patch should make use of the "port" local
variable. However, local variables are not available from inside plugin
patches, which are rendered outside of the context of the parent templates.
For a more extensive description of the problem, see:
https://github.com/overhangio/tutor-mfe/pull/23#issuecomment-964016190
We still want to make it easy for developers to decide what should the port be
for caddy hosts. To do so, we make use of environment variables that are passed
at runtime to the caddy container.
Thus, a regular plugin patch should look like this:
{{ PLUGIN_HOST }}{$default_site_port} {
import proxy "myplugin:8000"
}
Forum is an optional feature, and as such it deserves its own plugin. Starting
from Maple, users will be able to install the forum from
https://github.com/overhangio/tutor-forum/
Close #450.
In some cases, plugins declare an entry point but cannot be loaded.
This is the case when they depend on a version of tutor that is not the
one that is currently installed. This use case is very frequent when
working on multiple versions at the same time (i.e: right now, while we
are working on the Maple release). In such cases, it's best just to
ignore the plugin entirely rather than having to re-install all plugins
in the virtualenv.
- [Improvement] Upgrade all services to open-release/lilac.3.
- [Feature] Make it possible to override job configuration in
development: if they exist, `dev/docker-compose.jobs.yml` and
`dev/docker-compose.jobs.override.yml` will be loaded when running jobs.
- [Improvement] Faster `tutor local start` by building only necessary
images.
Previously, job declarations were always loaded from local/docker-compose.yml
and local/docker-compose.jobs.yml. This meant that it was not possible to
override job declarations in dev mode. It is now the case, with
dev/docker-compose.jobs.yml and dev/docker-compose.jobs.override.yml. Neither
of these files exist yet... But who knows? we might need this feature one day.
In any case the code is much cleaner now.
Before, custom `docker_compose_func` arguments had to be passed to job runners.
This was not very elegant. Also, it prevented us from loading custom job files
in development.
Here, we adopt a better object-oriented approach, where context classes are
ordered hierarchically.
This paves the way for loading `dev/docker-compose.jobs.yml` files in `tutor
dev init` commands -- which will be necessary to fix permissions in dev/local
mode.
Previously, we were building all images every time we ran a "local start"
command. This was causing unnecessary rebuild. Here, instead, we make use of
the `docker-compose up --build`. This means that only the required images will
be rebuilt.
Limits the memory chek to the 'local quickstart' command, makes error
handling more accurate and adds warning messages for some conditions.
Also adds a mention of this in troubleshooting.rst.
Adds a check in the 'local' command group that requires at least
4 GB of RAM to be allocated to Docker when running any of the
local subcommands on macOS. This addresses a common issue where
Docker's default setting (2 GB) causes startup to crash with
misleading error messages.
- 💥[Improvement] Change the `settheme` command such that, by default, a custom theme is assigned to the LMS and the CMS, both in production and development mode.
With this change, containers are no longer run as "root" but as unprivileged
users. This is necessary in some environments, notably some Kubernetes
clusters.
To make this possible, we need to manually fix bind-mounted volumes in
docker-compose. This is pretty much equivalent to the behaviour in Kubernetes,
where permissions are fixed at runtime if the volume owner is incorrect. Thus,
we have a consistent behaviour between docker-compose and Kubernetes.
We achieve this by bind-mounting some repos inside "*-permissions" services.
These services run as root user on docker-compose and will fix the required
permissions, as per build/permissions/setowner.sh These services simply do not
run on Kubernetes, where we don't rely on bind-mounted volumes. There, we make
use of Kubernete's built-in volume ownership feature.
With this change, we get rid of the "openedx-dev" Docker image, in the sense
that it no longer has its own Dockerfile. Instead, the dev image is now simply
a different target in the multi-layer openedx Docker image. This makes it much
faster to build the openedx-dev image.
Because we declare the APP_USER_ID in the dev/docker-compose.yml file, we need
to pass the user ID from the host there. The only way to achieve that is with a
tutor config variable. The downside of this approach is that the
dev/docker-compose.yml file is no longer portable from one machine to the next.
We consider that this is not such a big issue, as it affects the development
environment only.
We take this opportunity to replace the base image of the "forum" image. There
is now no need to re-install ruby inside the image. The total image size is
only decreased by 10%, but re-building the image is faster.
In order to run the smtp service as non-root, we switch from namshi/smtp to
devture/exim-relay. This change should be backward-compatible.
Note that the nginx container remains privileged. We could switch to
nginxinc/nginx-unprivileged, but it's probably not worth the effort, as we are
considering to get rid of the nginx container altogether.
Close #323.
Get Tutor to work on the master branches of Open edX. The corresponding images
will have to be rebuilt manually. Note that the process to contribute to the
nightly branch is slightly different from the master branch (see the
instructions from the corresponding tutorial).
In conversations with edX, we learned that the name "edge" had negative
undertones for historical reasons. Thus, we switch to "nightly", which means
pretty much the same thing.
Here, we make it possible to automatically append a suffix to the version and app
name (in the sense of appdirs). This guarantees that a tutor edge project will
not accidentally override another community release.
In addition, we take the opportunity to document the tutor versioning format.
(I've been meaning to do that for a long time)
Previously, the list of domain names to which a theme was assigned had to be
specified manually. Now, the themes are automatically assigned to the LMS and
the CMS, both in development and production modes.
See: https://discuss.overhang.io/t/no-activation-email-errors-logged-on-user-sign-up/1969
A 500 error was being triggered during user registration.
Traceback (most recent call last):
File "/openedx/venv/lib/python3.8/site-packages/django/core/handlers/exception.py", line 34, in inner
response = get_response(request)
File "/openedx/venv/lib/python3.8/site-packages/django/core/handlers/base.py", line 115, in _get_response
response = self.process_exception_by_middleware(e, request)
File "/openedx/venv/lib/python3.8/site-packages/django/core/handlers/base.py", line 113, in _get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/openedx/venv/lib/python3.8/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
return view_func(*args, **kwargs)
File "/openedx/venv/lib/python3.8/site-packages/django/views/generic/base.py", line 71, in view
return self.dispatch(request, *args, **kwargs)
File "/openedx/venv/lib/python3.8/site-packages/django/utils/decorators.py", line 45, in _wrapper
return bound_method(*args, **kwargs)
File "/openedx/venv/lib/python3.8/site-packages/django/utils/decorators.py", line 45, in _wrapper
return bound_method(*args, **kwargs)
File "/openedx/venv/lib/python3.8/site-packages/django/views/decorators/debug.py", line 76, in sensitive_post_parameters_wrapper
return view(request, *args, **kwargs)
File "./openedx/core/djangoapps/user_authn/views/register.py", line 485, in dispatch
return super().dispatch(request, *args, **kwargs)
File "/openedx/venv/lib/python3.8/site-packages/rest_framework/views.py", line 509, in dispatch
response = self.handle_exception(exc)
File "/openedx/venv/lib/python3.8/site-packages/rest_framework/views.py", line 469, in handle_exception
self.raise_uncaught_exception(exc)
File "/openedx/venv/lib/python3.8/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception
raise exc
File "/openedx/venv/lib/python3.8/site-packages/rest_framework/views.py", line 506, in dispatch
response = handler(request, *args, **kwargs)
File "/openedx/venv/lib/python3.8/site-packages/django/utils/decorators.py", line 45, in _wrapper
return bound_method(*args, **kwargs)
File "/openedx/venv/lib/python3.8/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
return view_func(*args, **kwargs)
File "/openedx/venv/lib/python3.8/site-packages/django/utils/decorators.py", line 45, in _wrapper
return bound_method(*args, **kwargs)
File "/openedx/venv/lib/python3.8/site-packages/ratelimit/decorators.py", line 24, in _wrapped
return fn(request, *args, **kw)
File "./openedx/core/djangoapps/user_authn/views/register.py", line 529, in post
response, user = self._create_account(request, data)
File "./openedx/core/djangoapps/user_authn/views/register.py", line 572, in _create_account
user = create_account_with_params(request, data)
File "./openedx/core/djangoapps/user_authn/views/register.py", line 236, in create_account_with_params
compose_and_send_activation_email(user, profile, registration)
File "/openedx/edx-platform/common/djangoapps/student/views/management.py", line 214, in compose_and_send_activation_email
send_activation_email.delay(str(msg))
File "/openedx/venv/lib/python3.8/site-packages/edx_ace/serialization.py", line 29, in __str__
return json.dumps(self, cls=MessageEncoder)
File "/opt/pyenv/versions/3.8.6/lib/python3.8/json/__init__.py", line 234, in dumps
return cls(
File "/opt/pyenv/versions/3.8.6/lib/python3.8/json/encoder.py", line 199, in encode
chunks = self.iterencode(o, _one_shot=True)
File "/opt/pyenv/versions/3.8.6/lib/python3.8/json/encoder.py", line 257, in iterencode
return _iterencode(o, 0)
File "/openedx/venv/lib/python3.8/site-packages/edx_ace/serialization.py", line 119, in default
return super().default(o) # pragma: no cover
File "/opt/pyenv/versions/3.8.6/lib/python3.8/json/encoder.py", line 179, in default
raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type LazyStaticAbsoluteUrl is not JSON serializable
2021-09-28 05:21:52,174 ERROR 122 [django.request] [user 11] [ip XY.XY.XY.XY] log.py:222 - Internal Server Error: /api/user/v2/account/registration/
Traceback (most recent call last):
File "/openedx/venv/lib/python3.8/site-packages/django/core/handlers/exception.py", line 34, in inner
response = get_response(request)
File "/openedx/venv/lib/python3.8/site-packages/django/core/handlers/base.py", line 115, in _get_response
response = self.process_exception_by_middleware(e, request)
File "/openedx/venv/lib/python3.8/site-packages/django/core/handlers/base.py", line 113, in _get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/openedx/venv/lib/python3.8/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
return view_func(*args, **kwargs)
File "/openedx/venv/lib/python3.8/site-packages/django/views/generic/base.py", line 71, in view
return self.dispatch(request, *args, **kwargs)
File "/openedx/venv/lib/python3.8/site-packages/django/utils/decorators.py", line 45, in _wrapper
return bound_method(*args, **kwargs)
File "/openedx/venv/lib/python3.8/site-packages/django/utils/decorators.py", line 45, in _wrapper
return bound_method(*args, **kwargs)
File "/openedx/venv/lib/python3.8/site-packages/django/views/decorators/debug.py", line 76, in sensitive_post_parameters_wrapper
return view(request, *args, **kwargs)
File "./openedx/core/djangoapps/user_authn/views/register.py", line 485, in dispatch
return super().dispatch(request, *args, **kwargs)
File "/openedx/venv/lib/python3.8/site-packages/rest_framework/views.py", line 509, in dispatch
response = self.handle_exception(exc)
File "/openedx/venv/lib/python3.8/site-packages/rest_framework/views.py", line 469, in handle_exception
self.raise_uncaught_exception(exc)
File "/openedx/venv/lib/python3.8/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception
raise exc
File "/openedx/venv/lib/python3.8/site-packages/rest_framework/views.py", line 506, in dispatch
response = handler(request, *args, **kwargs)
File "/openedx/venv/lib/python3.8/site-packages/django/utils/decorators.py", line 45, in _wrapper
return bound_method(*args, **kwargs)
File "/openedx/venv/lib/python3.8/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
return view_func(*args, **kwargs)
File "/openedx/venv/lib/python3.8/site-packages/django/utils/decorators.py", line 45, in _wrapper
return bound_method(*args, **kwargs)
File "/openedx/venv/lib/python3.8/site-packages/ratelimit/decorators.py", line 24, in _wrapped
return fn(request, *args, **kw)
File "./openedx/core/djangoapps/user_authn/views/register.py", line 529, in post
response, user = self._create_account(request, data)
File "./openedx/core/djangoapps/user_authn/views/register.py", line 572, in _create_account
user = create_account_with_params(request, data)
File "./openedx/core/djangoapps/user_authn/views/register.py", line 236, in create_account_with_params
compose_and_send_activation_email(user, profile, registration)
File "/openedx/edx-platform/common/djangoapps/student/views/management.py", line 214, in compose_and_send_activation_email
send_activation_email.delay(str(msg))
File "/openedx/venv/lib/python3.8/site-packages/edx_ace/serialization.py", line 29, in __str__
return json.dumps(self, cls=MessageEncoder)
File "/opt/pyenv/versions/3.8.6/lib/python3.8/json/__init__.py", line 234, in dumps
return cls(
File "/opt/pyenv/versions/3.8.6/lib/python3.8/json/encoder.py", line 199, in encode
chunks = self.iterencode(o, _one_shot=True)
File "/opt/pyenv/versions/3.8.6/lib/python3.8/json/encoder.py", line 257, in iterencode
return _iterencode(o, 0)
File "/openedx/venv/lib/python3.8/site-packages/edx_ace/serialization.py", line 119, in default
return super().default(o) # pragma: no cover
File "/opt/pyenv/versions/3.8.6/lib/python3.8/json/encoder.py", line 179, in default
raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type LazyStaticAbsoluteUrl is not JSON serializable
The reason for that was that edx-ace could not json-serialize the context to be
passed to the registration email renderer. That was caused by the
LazyStaticAbsoluteUrl object created to address missing logo in registration
email. To make sure that this object is serializable by
edx_ace.serialization.MessageEncoder, we add a trivial .to_json() method to the
LazyStaticAbsoluteUrl class.
This error could (at first) not be reproduced in development, because
AUTOMATIC_AUTH_FOR_TESTING is set to true in devstack settings.
When upgrading mongodb, the mongodb container takes a little while to become
ready. Running the "exec" command thus triggers an error:
docker-compose -f /path/to/env/local/docker-compose.yml -f /path/to/env/local/docker-compose.prod.yml --project-name tutor_local exec mongodb mongo --eval db.adminCommand({ setF
eatureCompatibilityVersion: "4.0" })
MongoDB shell version v4.0.24
connecting to: mongodb://127.0.0.1:27017/?gssapiServiceName=mongodb
2021-06-14T10:53:21.510+0000 E QUERY [js] Error: couldn't connect to server 127.0.0.1:27017, connection attempt failed: SocketException: Error connecting to 127.0.0.1:27017 :: caused by :: Connection refused:
connect@src/mongo/shell/mongo.js:356:17
@(connect):2:6
exception: connect failed
Error: Command failed with status 1: docker-compose -f /path/to/env/local/docker-compose.yml -f /path/to/env/local/docker-compose.prod.yml --project-name tutor_local exec mongodb mongo --eval db.adminCommand({ setFeatureCompatibilityVersion: "4.0" })
We add a "sleep" statement to the upgrade process to ensure that the mongodb
container is available.
Turns out, the authentication mechanism should only be defined if there
is an actual authentication. For now, because of the urgency, we
hardcode this auth_mech to ":scram". We'll add a way to override it if
necessary, in the future.
- [Improvement] Make it easier to run edx-platform unit tests.
- [Bugfix] Fix segmentation fault during `tutor config save` on Mac OS M1 (#473). Thanks @ghassanmas!
- [Bugfix] Fix a bug that prevented connecting to external MongoDB instances.
- [Improvement] Make sure that the logo included in email notifications (including discussion responses) is the same as the site logo.
- [Bugfix] Install IPython directly from pypi instead of installing it from source (the reason it was installed from source is no longer relevant). The effect of this shall speed up the process of building the openedx-dev Docker image.
- [Feature] Add "openedx-dockerfile-post-git-checkout" patch.
- [Improvement] In the "openedx" Docker images, convert git patches to cherry-picks for a cleaner source tree.
- 💥[Feature] Make it possible to override local job configuration. This deprecates the older model for running jobs which dates back from a long time ago.
It should be unnecessary to build a custom openedx-dev Docker image. All tests
can run from within the dev Docker image, with a couple additional environment
variables.
Previously, the logo included in emails was loaded from edX' CDN. Here, we make
sure that the logo is actually the same as the site logo. Because the logo may
be theme-specific, we need to compute the logo url at runtime, and use a
lazily-evaluated string.
Close #447.
Change the source of installation Ipyton, install it from PyPy instead of source, which shall speed up the process of building the docker image for openedx-dev
With "git patch", the resulting source tree was dirty, showing uncommitted
changes. Here, we replace "git patch" with "git cherry-pick". We avoid pulling
the entire remote repo by fetching individual commits. To do that, we need to
assign an identity to the git user.
- [Improvement] Upgrade all services to open-release/lilac.2.
- [Bugfix] Fix "`sh` is not a recognized command" in some plugins, including minio.
- [Improvement] Set the default contact mailing email address
- [Bugfix] Fix minio initialisation in Kubernetes.
- [Bugfix] Fix "Invalid command argument" during upgrade from Koa to Lilac.
- [Bugfix] Fix mysql initialisation in docker-compose==2.0.0beta4.
- [Improvement] Tutor is now published on pypi as "tutor".
An issue with the latest release of docker-compose was reported here:
https://discuss.overhang.io/t/undefined-entrypoint-throws-error-in-docker-compose-2-0-0-beta-4/1716
The mysql-job definition had an empty entrypoint (`[]`). This was causing the following error:
the initiation of mysql fails with “services.mysql-job.entrypoint must be a string …
Error: Command failed with status 15”
I can't remember at all why we had to define an empty entrypoint. It probably
has to do with the fact that we could not run `sh -e -c "..."` commands in
mysql jobs. Similarly, the k8s job definition sets `command: []`. I tested both
local and k8s deployments without these definitions and they work just fine. So
I guess we can get rid of them.
- [Bugfix] Fix double pulling mongodb image when upgrading from Koa to Lilac.
- [Improvement] Better logging during `plugins disable`.
- [Bugfix] Fix "upstream sent too big header" error during login of existing users after a Koa to Lilac upgrade.
- [Feature] Added the ability to skip `config.yml` file modification while running `tutor config save` command with `-e` or `--env-only` flag.
- [Feature] Add new config value `FORUM_MONGODB_DATABASE` to set the forum database name
In config.yml the new value FORUM_MONGO_DB_DATABASE was added with `cs_comments_service` as default value.
In docker-entrypoint.sh of forum I changed the hardcoded `cs_commecnts_service` with the new config value.
Multiple .yml files changed to handle the new config value.
When disable a plugin that set config entried, such as the minio plugin, tutor was logging the following:
Disabling plugin minio...
Removed config entry OPENEDX_AWS_ACCESS_KEY=openedx
Removed config entry OPENEDX_AWS_SECRET_ACCESS_KEY={{ MINIO_AWS_SECRET_ACCESS_KEY }}
Plugin disabled
The config values were not rendered during printing, which is a shame, because
the whole point of this log line is to warn users of passwords/secrets that are
being removed. Here, we make sure that the config values are properly rendered.
The new logs are now:
Disabling plugin minio...
Removing config entry OPENEDX_AWS_ACCESS_KEY=openedx
Removing config entry OPENEDX_AWS_SECRET_ACCESS_KEY=64vpCVLxhDxBuNjakSrX4CQg
Plugin disabled
- [Improvement] Avoid permission issues in Kubernetes/Openshift for users who do not have the rights to edit their namespace.
- [Improvement] Better Kubernetes object creation.
In most cases, it makes very little sense to edit the namespace that an
application is running in. Quite often, users are granted access to just one
namespace and don't have the necessary rights to edit the namespace -- and for
good security reasons. In such cases, the k8s namespace object already exists
and there is no need for the user to edit or create it. Here, what we do is
that we create the namespace only if it does not exist. This should solve quite
a few permission issues, notably for Openshift users.
- 💥[Security] Disable python-evaluated input by default as we don't run codejail.
- [Bugfix] Fix missing discussion notifications.
- [Improvement] Better error logging when loading a plugin from an incompatible version.
We were forced to pin click to < v8 because of missing dependencies. In
particular, click_repl was broken. This is no longer the case, as click_repl
0.20 was published. Also, Jinja2 now includes type annotations, which allows us
to get rid of a few "# type: ignore" statements.
We take the opportunity to upgrade all requirements, which allows us resolve a
security issue on urllib3<1.26.0.
As described in issue #284, tutor does not come with codejail enabled out of
the box. Actually, we don't even have a working plugin, yet. To prevent users
from running unsafe code, we explicitely disable python-evaluated input by
disabling the "python" interpreter. This might break some courses; thus, this
is a non-backward compatible change.
Discussion email notification relies on edx_ace and celery routing
key for ace is not set properly by default, which causes the send
mail task is never consumed.
Close #439.
When upgrading Tutor plugins to the next release, I often end up with a
virtualenv that contains plugins that depend on different versions of
tutor-openedx. This causes a crash that did not log the name of the responsible
package. For instance:
Traceback (most recent call last):
File "/home/regis/venvs/tutor/bin/tutor", line 11, in <module>
load_entry_point('tutor-openedx', 'console_scripts', 'tutor')()
File "/home/regis/projets/overhang/repos/overhang/tutor/tutor/commands/cli.py", line 37, in main
add_plugin_commands(cli)
File "/home/regis/projets/overhang/repos/overhang/tutor/tutor/commands/plugins.py", line 137, in add_plugin_commands
for plugin in plugins.iter_installed():
File "/home/regis/projets/overhang/repos/overhang/tutor/tutor/plugins.py", line 401, in iter_installed
yield from Plugins.iter_installed()
File "/home/regis/projets/overhang/repos/overhang/tutor/tutor/plugins.py", line 362, in iter_installed
for plugin in PluginClass.iter_installed():
File "/home/regis/projets/overhang/repos/overhang/tutor/tutor/plugins.py", line 199, in iter_installed
for plugin in cls.iter_load():
File "/home/regis/projets/overhang/repos/overhang/tutor/tutor/plugins.py", line 233, in iter_load
yield cls(entrypoint)
File "/home/regis/projets/overhang/repos/overhang/tutor/tutor/plugins.py", line 220, in __init__
super().__init__(entrypoint.name, entrypoint.load())
File "/home/regis/venvs/tutor/lib/python3.8/site-packages/pkg_resources/__init__.py", line 2442, in load
self.require(*args, **kwargs)
File "/home/regis/venvs/tutor/lib/python3.8/site-packages/pkg_resources/__init__.py", line 2465, in require
items = working_set.resolve(reqs, env, installer, extras=self.extras)
File "/home/regis/venvs/tutor/lib/python3.8/site-packages/pkg_resources/__init__.py", line 791, in resolve
raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.VersionConflict: (tutor-openedx 11.2.11 (/home/regis/projets/overhang/repos/overhang/tutor), Requirement.parse('tutor-openedx<13.0.0,>=12.0.0'))
In this commit, we introduce an error log that displays the name and location
of the package. E.g:
Failed to load entrypoint 'minio = tutorminio.plugin' from distribution tutor-minio 12.0.0
- [Security] Apply Django security patches by upgrading from 2.2.20 to 2.2.23.
- [Bugfix] Fix video unit completion (see [pull
request](https://github.com/edx/edx-platform/pull/27230)).
- [Bugfix] Fix crashing installation because of a major release of all
Pallets projects.
- [Bugfix] Fix crash in `local quickstart -p` command.
- [Bugfix] Fix 502 error on request to lms with header larger than the
maximum uwsgi buffer size.
When running `tutor local quickstart -p` we were getting the following error:
Usage: custom [OPTIONS] ARGS...
Try 'custom --help' for help.
Error: Missing argument 'ARGS...'.
The docker-compose command sometimes accept a single command ("pull") with zero
argument.
See: https://discuss.overhang.io/t/local-quickstart-not-working-when-pullimages-enabled/1526
According to [1], request to uwsgi with header
larger than the default value 4096 will end up
with below error as #426:
invalid request block size: 4123 (max 4096)...skip
hr_instance_read(): Connection reset by peer
[plugins/http/http.c line 647]
This commit fixes it by changing the maximum
buffer size as 4096
Also: Similar issue was already identifed and
fixed in [2] on tutor-ecommerce
[1] https://uwsgi-docs.readthedocs.io/en/latest/ThingsToKnow.html
[2] 6df2c99362
Close #426
I stumbled upon a bug that should have been detected by the type
checking. Turns out, considering that config is of type Dict[str, Any]
means that we can use just any method on all config values -- which is
terrible. I discovered this after I set `config["PLUGINS"] = None`:
this triggered a crash when I enabled a plugin.
We resolve this by making the Config type more explicit. We also take
the opportunity to remove a few cast statements.
During Docker images build process, apply custom edx-platform patches when tutor patch 'openedx-dockerfile-git-patches-default' is defined or apply current release patches in other case. It avoids possible conflicts between the actually used edx-platform version and the current release patches.
- [Improvement] Upgrade all services to open-release/koa.3.
- [Feature] Make it possible to build the openedx Docker image with a custom openedx-i18n version with the ``--build-arg OPENEDX_I18N_VERSION=custom`` command line argument.
We remove security patches and custom fixes which are now part of koa.3.
We take the opportunity to make it possible to build the openedx Docker image
without relying on a corresponding openedx-i18n repo tag: often, we want to
test whether the image simply builds successfully, and we don't need up-to-date
translations. For those cases, it's now possible to pass the `-a
OPENEDX_I18N_VERSION=oldertag` build argument.
When the PLUGINS config entry is None (`PLUGINS:`), the following error
was being triggered:
File "/.../tutor/tutor/plugins.py",
line 304, in is_enabled
return name in config.get(CONFIG_KEY, [])
TypeError: argument of type 'NoneType' is not iterable
- [Bugfix] Fix "Invalid Request" error during SAML authentication
(thanks @Ant1x!).
- [Feature] Add `images build --pull` option to update base images.
- [Improvement] Annotate types all over the Tutor code base.
- [Bugfix] Fix parsing of YAML CLI arguments that include equal "="
signs.
- [Bugfix] Fix minor edge case in `long_to_base64` utility function.
- [Improvement] Add openedx patches to add settings during build
process.
This option is mostly useless to us in CI, as it attempts (and fails) to
update the base image when building the dev image. For good reason: the
base image of the dev image is the latest openedx, which has not been
pushed to docker.io at the time of building -_-
Annotations were generated with pyannotate:
https://github.com/dropbox/pyannotate
We are running in strict mode, which is awesome!
This affects a large part of the code base, which might be an issue for
people running a fork of Tutor. Nonetheless, the behavior should not be
affected. If anything, this process has helped find and resolve a few
type-related bugs. Thus, this is not considered as a breaking change.
It made little sense to create dedicated context classes for local/dev.
Instead, we create local/dev compose methods which can be easily reused.
Also, we renamed the "scripts" module to better reflect its function.
lms-worker was configured to run CMS tasks instead of LMS tasks. I'm not
sure what tasks were being dismissed, and what is the actual production
impact.
Redis data was not actually persisted, because the redis configuration file was
not mounted from the right location. In order to mount redis data in a
host-mounted directory, the working directory has to be properly set.
The problem was occurring both with docker-compose and Kubernetes.
Close #404.
When tutor is run as root, the `tutor config printroot` and `tutor
config printvalue` commands used to write to stdout, which was causing
issues when combined with `tutor config render` and `tutor local
settheme` commands.
See discussion:
https://discuss.overhang.io/t/error-while-install-indigo-theme/1056
First, allow using custom Django settings on a development
environment (as documented but not implemented), setting it to the
correct value of `tutor.development`. Prior to this, `tutor dev
runserver lms` would default to `tutor.production` when on a custom edX
branch.
Second, fix the documentation so the correct environment variable is
described, at the same time removing an option that doesn't seem to work.
See discussion: https://discuss.overhang.io/t/koa-dev-lms-doesnt-find-static-content/1250
