upgrading.
- [Bugfix] During upgrade, make sure that environment is up-to-date
prior to prompting to rebuild the custom images.
- [Bugfix] Fix ownership of mysql data, in particular when upgrading a
Kubernetes cluster to Maple.
- [Bugfix] Ensure that ``tutor k8s upgrade`` is run during ``tutor k8s
quickstart``, when necessary.
- 💥[Bugfix] By default, detect the current version during ``tutor
k8s/local upgrade``.
- [Bugfix] Fix upgrading from Lilac to Maple on Kubernetes by deleting
deployments and services.
`upgrade` had several issues, which are summarized here:
https://discuss.overhang.io/t/confusing-instructions-during-upgrade/2281/7
- The docs say that you should run quickstart, but what most people will see is
the big command tutor local upgrade --from=lilac verbatim paragraph.
- The local upgrade command should be very explicit about the fact that users
need to run quickstart.
- Maybe the name of the local upgrade command should be improved.
- When upgrading tutor from one major release to the next, there should be a
more explicit warning to inform users of what they are doing (see this other
conversation 1)
- We should tell people that they almost certainly need to enable the tutor and
the mfe plugins, if they are not enabled during upgrade.
- A link to all of the breaking changes from the changelog should be
prominently displayed during upgrade.
- The docs should emphasize that upgrading from one major release to the next
is potentially a risky endeavor and that downgrading is not possible. The docs
should also link to the changelog.
This commit has grown slightly beyond the intended scope, but the changes should be mostly positive.
In theory, we can assign ownership of mysql data to just any user. But in
Lilac, mysql was running with user 999. When upgrading to Maple, on Kubernetes,
the fsGroupChangePolicy was causing a change of the data *group* (to 1000) but
not of the user. This was causing a crash with the following error:
[ERROR] InnoDB: The error means mysqld does not have the access rights to the directory.
When upgrading from Lilac, all services break with the following error:
Service "***" is invalid: spec.ports[0].nodePort: Forbidden: may not be used when `type` is 'ClusterIP'
Upgrading deployments fails as well:
Deployment.apps "***" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{"app.kubernetes.io/instance":"openedx-********", "app.kubernetes.io/managed-by":"tutor", "app.kubernetes.io/name":"***", "app.kubernetes.io/part-of":"openedx"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable
That's because deployments and services need to be deleted as part of the Maple
upgrade. So that's what we do as part of `tutor k8s upgrade --from=lilac`. And
we take the opportunity to:
1. Run upgrade as part of quickstart, when necessary.
2. Default to lilac during `tutor k8s upgrade`.
Close #551.
- [Security] Upgrade Django to 3.2.11 in edx-platform.
- [Security] Prevent non-staff users from searching usernames by email by
abusing the logout url.
Without this patch, it is possible to search for account info including
username by using the email of a learner. This fix disallows searching using
email by regular users and restricts this feature to only staff and superusers.
- A shared cookie domain between lms and cms is no longer recommended:
https://github.com/edx/edx-platform/blob/master/docs/guides/studio_oauth.rst
- refactor: clean mounted data folder in lms/cms. In Lilac, the
bind-mounted lms/data and cms/data folders are a mess because new
folders are created there for every new course organisation. These
folders are empty. As far as we know they are useless... With this
change we move these folders to a dedicated "modulestore" subdirectory;
which corresponds better to the initial intent of the fs_root setting.
- fix: frontend failure during login to the lms. See:
https://github.com/openedx/build-test-release-wg/issues/104
- feat: move all forum-related code to a dedicated plugin. Forum is an
optional feature, and as such it deserves its own plugin. Starting from
Maple, users will be able to install the forum from
https://github.com/overhangio/tutor-forum/
- migrate from DCS_* session cookie settings to SESSION_*. That's
because edx-platform no longer depends on django-cookies-samesite. Close
https://github.com/openedx/build-test-release-wg/issues/110
- get rid of tons of deprecation warnings in the lms/cms
- feat: make it possible to point to themed assets. Cherry-picking this
change makes it possible to point to themed assets with a theme-agnostic
url, notably from MFEs.
- Install all official plugins as part of the `tutor[full]` package.
- Don't print error messages about loading plugins during autocompletion.
- Prompt for image building when upgrading from one release to the next.
- Add `tutor local start --skip-build` option to skip building Docker images.
Close #450.
Close #545.
- 💥[Improvement] Fail on incorrect image name argument in `images
build/pull/push/printtag` commands.
- [Bugfix] Remove trailing slashes in docker-compose files for
[compatibility with docker-compose v2 in
WSL](https://github.com/docker/compose/issues/8558).
- [Improvement] `settheme` now works with preview domain.
- [Feature] Allow specifying extra pip packages through config.yml.
Added OPENEDX_EXTRA_PIP_REQUIREMENTS setting, which allows to specify
extra pip packages that should be installed.
Moved "openedx-scorm-xblock" package from Dockerfile to the new setting
in the config.yml.
When running "tutor k8s" we can disable the forum deployment and
service by setting RUN_FORUM to false, but the corresponding job is
added to jobs.yml unconditionally.
Add a conditional to tutor/templates/k8s/jobs.yml so that the forum
job definition is only included when RUN_FORUM is true.
Fixes #525.
In some cases, plugins declare an entry point but cannot be loaded.
This is the case when they depend on a version of tutor that is not the
one that is currently installed. This use case is very frequent when
working on multiple versions at the same time (i.e: right now, while we
are working on the Maple release). In such cases, it's best just to
ignore the plugin entirely rather than having to re-install all plugins
in the virtualenv.
- [Improvement] Upgrade all services to open-release/lilac.3.
- [Feature] Make it possible to override job configuration in
development: if they exist, `dev/docker-compose.jobs.yml` and
`dev/docker-compose.jobs.override.yml` will be loaded when running jobs.
- [Improvement] Faster `tutor local start` by building only necessary
images.
Previously, job declarations were always loaded from local/docker-compose.yml
and local/docker-compose.jobs.yml. This meant that it was not possible to
override job declarations in dev mode. It is now the case, with
dev/docker-compose.jobs.yml and dev/docker-compose.jobs.override.yml. Neither
of these files exist yet... But who knows? we might need this feature one day.
In any case the code is much cleaner now.
Previously, we were building all images every time we ran a "local start"
command. This was causing unnecessary rebuild. Here, instead, we make use of
the `docker-compose up --build`. This means that only the required images will
be rebuilt.
Limits the memory chek to the 'local quickstart' command, makes error
handling more accurate and adds warning messages for some conditions.
Also adds a mention of this in troubleshooting.rst.
Adds a check in the 'local' command group that requires at least
4 GB of RAM to be allocated to Docker when running any of the
local subcommands on macOS. This addresses a common issue where
Docker's default setting (2 GB) causes startup to crash with
misleading error messages.
- 💥[Improvement] Change the `settheme` command such that, by default, a custom theme is assigned to the LMS and the CMS, both in production and development mode.
Previously, the list of domain names to which a theme was assigned had to be
specified manually. Now, the themes are automatically assigned to the LMS and
the CMS, both in development and production modes.
See: https://discuss.overhang.io/t/no-activation-email-errors-logged-on-user-sign-up/1969
A 500 error was being triggered during user registration.
Traceback (most recent call last):
File "/openedx/venv/lib/python3.8/site-packages/django/core/handlers/exception.py", line 34, in inner
response = get_response(request)
File "/openedx/venv/lib/python3.8/site-packages/django/core/handlers/base.py", line 115, in _get_response
response = self.process_exception_by_middleware(e, request)
File "/openedx/venv/lib/python3.8/site-packages/django/core/handlers/base.py", line 113, in _get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/openedx/venv/lib/python3.8/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
return view_func(*args, **kwargs)
File "/openedx/venv/lib/python3.8/site-packages/django/views/generic/base.py", line 71, in view
return self.dispatch(request, *args, **kwargs)
File "/openedx/venv/lib/python3.8/site-packages/django/utils/decorators.py", line 45, in _wrapper
return bound_method(*args, **kwargs)
File "/openedx/venv/lib/python3.8/site-packages/django/utils/decorators.py", line 45, in _wrapper
return bound_method(*args, **kwargs)
File "/openedx/venv/lib/python3.8/site-packages/django/views/decorators/debug.py", line 76, in sensitive_post_parameters_wrapper
return view(request, *args, **kwargs)
File "./openedx/core/djangoapps/user_authn/views/register.py", line 485, in dispatch
return super().dispatch(request, *args, **kwargs)
File "/openedx/venv/lib/python3.8/site-packages/rest_framework/views.py", line 509, in dispatch
response = self.handle_exception(exc)
File "/openedx/venv/lib/python3.8/site-packages/rest_framework/views.py", line 469, in handle_exception
self.raise_uncaught_exception(exc)
File "/openedx/venv/lib/python3.8/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception
raise exc
File "/openedx/venv/lib/python3.8/site-packages/rest_framework/views.py", line 506, in dispatch
response = handler(request, *args, **kwargs)
File "/openedx/venv/lib/python3.8/site-packages/django/utils/decorators.py", line 45, in _wrapper
return bound_method(*args, **kwargs)
File "/openedx/venv/lib/python3.8/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
return view_func(*args, **kwargs)
File "/openedx/venv/lib/python3.8/site-packages/django/utils/decorators.py", line 45, in _wrapper
return bound_method(*args, **kwargs)
File "/openedx/venv/lib/python3.8/site-packages/ratelimit/decorators.py", line 24, in _wrapped
return fn(request, *args, **kw)
File "./openedx/core/djangoapps/user_authn/views/register.py", line 529, in post
response, user = self._create_account(request, data)
File "./openedx/core/djangoapps/user_authn/views/register.py", line 572, in _create_account
user = create_account_with_params(request, data)
File "./openedx/core/djangoapps/user_authn/views/register.py", line 236, in create_account_with_params
compose_and_send_activation_email(user, profile, registration)
File "/openedx/edx-platform/common/djangoapps/student/views/management.py", line 214, in compose_and_send_activation_email
send_activation_email.delay(str(msg))
File "/openedx/venv/lib/python3.8/site-packages/edx_ace/serialization.py", line 29, in __str__
return json.dumps(self, cls=MessageEncoder)
File "/opt/pyenv/versions/3.8.6/lib/python3.8/json/__init__.py", line 234, in dumps
return cls(
File "/opt/pyenv/versions/3.8.6/lib/python3.8/json/encoder.py", line 199, in encode
chunks = self.iterencode(o, _one_shot=True)
File "/opt/pyenv/versions/3.8.6/lib/python3.8/json/encoder.py", line 257, in iterencode
return _iterencode(o, 0)
File "/openedx/venv/lib/python3.8/site-packages/edx_ace/serialization.py", line 119, in default
return super().default(o) # pragma: no cover
File "/opt/pyenv/versions/3.8.6/lib/python3.8/json/encoder.py", line 179, in default
raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type LazyStaticAbsoluteUrl is not JSON serializable
2021-09-28 05:21:52,174 ERROR 122 [django.request] [user 11] [ip XY.XY.XY.XY] log.py:222 - Internal Server Error: /api/user/v2/account/registration/
Traceback (most recent call last):
File "/openedx/venv/lib/python3.8/site-packages/django/core/handlers/exception.py", line 34, in inner
response = get_response(request)
File "/openedx/venv/lib/python3.8/site-packages/django/core/handlers/base.py", line 115, in _get_response
response = self.process_exception_by_middleware(e, request)
File "/openedx/venv/lib/python3.8/site-packages/django/core/handlers/base.py", line 113, in _get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/openedx/venv/lib/python3.8/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
return view_func(*args, **kwargs)
File "/openedx/venv/lib/python3.8/site-packages/django/views/generic/base.py", line 71, in view
return self.dispatch(request, *args, **kwargs)
File "/openedx/venv/lib/python3.8/site-packages/django/utils/decorators.py", line 45, in _wrapper
return bound_method(*args, **kwargs)
File "/openedx/venv/lib/python3.8/site-packages/django/utils/decorators.py", line 45, in _wrapper
return bound_method(*args, **kwargs)
File "/openedx/venv/lib/python3.8/site-packages/django/views/decorators/debug.py", line 76, in sensitive_post_parameters_wrapper
return view(request, *args, **kwargs)
File "./openedx/core/djangoapps/user_authn/views/register.py", line 485, in dispatch
return super().dispatch(request, *args, **kwargs)
File "/openedx/venv/lib/python3.8/site-packages/rest_framework/views.py", line 509, in dispatch
response = self.handle_exception(exc)
File "/openedx/venv/lib/python3.8/site-packages/rest_framework/views.py", line 469, in handle_exception
self.raise_uncaught_exception(exc)
File "/openedx/venv/lib/python3.8/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception
raise exc
File "/openedx/venv/lib/python3.8/site-packages/rest_framework/views.py", line 506, in dispatch
response = handler(request, *args, **kwargs)
File "/openedx/venv/lib/python3.8/site-packages/django/utils/decorators.py", line 45, in _wrapper
return bound_method(*args, **kwargs)
File "/openedx/venv/lib/python3.8/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
return view_func(*args, **kwargs)
File "/openedx/venv/lib/python3.8/site-packages/django/utils/decorators.py", line 45, in _wrapper
return bound_method(*args, **kwargs)
File "/openedx/venv/lib/python3.8/site-packages/ratelimit/decorators.py", line 24, in _wrapped
return fn(request, *args, **kw)
File "./openedx/core/djangoapps/user_authn/views/register.py", line 529, in post
response, user = self._create_account(request, data)
File "./openedx/core/djangoapps/user_authn/views/register.py", line 572, in _create_account
user = create_account_with_params(request, data)
File "./openedx/core/djangoapps/user_authn/views/register.py", line 236, in create_account_with_params
compose_and_send_activation_email(user, profile, registration)
File "/openedx/edx-platform/common/djangoapps/student/views/management.py", line 214, in compose_and_send_activation_email
send_activation_email.delay(str(msg))
File "/openedx/venv/lib/python3.8/site-packages/edx_ace/serialization.py", line 29, in __str__
return json.dumps(self, cls=MessageEncoder)
File "/opt/pyenv/versions/3.8.6/lib/python3.8/json/__init__.py", line 234, in dumps
return cls(
File "/opt/pyenv/versions/3.8.6/lib/python3.8/json/encoder.py", line 199, in encode
chunks = self.iterencode(o, _one_shot=True)
File "/opt/pyenv/versions/3.8.6/lib/python3.8/json/encoder.py", line 257, in iterencode
return _iterencode(o, 0)
File "/openedx/venv/lib/python3.8/site-packages/edx_ace/serialization.py", line 119, in default
return super().default(o) # pragma: no cover
File "/opt/pyenv/versions/3.8.6/lib/python3.8/json/encoder.py", line 179, in default
raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type LazyStaticAbsoluteUrl is not JSON serializable
The reason for that was that edx-ace could not json-serialize the context to be
passed to the registration email renderer. That was caused by the
LazyStaticAbsoluteUrl object created to address missing logo in registration
email. To make sure that this object is serializable by
edx_ace.serialization.MessageEncoder, we add a trivial .to_json() method to the
LazyStaticAbsoluteUrl class.
This error could (at first) not be reproduced in development, because
AUTOMATIC_AUTH_FOR_TESTING is set to true in devstack settings.
When upgrading mongodb, the mongodb container takes a little while to become
ready. Running the "exec" command thus triggers an error:
docker-compose -f /path/to/env/local/docker-compose.yml -f /path/to/env/local/docker-compose.prod.yml --project-name tutor_local exec mongodb mongo --eval db.adminCommand({ setF
eatureCompatibilityVersion: "4.0" })
MongoDB shell version v4.0.24
connecting to: mongodb://127.0.0.1:27017/?gssapiServiceName=mongodb
2021-06-14T10:53:21.510+0000 E QUERY [js] Error: couldn't connect to server 127.0.0.1:27017, connection attempt failed: SocketException: Error connecting to 127.0.0.1:27017 :: caused by :: Connection refused:
connect@src/mongo/shell/mongo.js:356:17
@(connect):2:6
exception: connect failed
Error: Command failed with status 1: docker-compose -f /path/to/env/local/docker-compose.yml -f /path/to/env/local/docker-compose.prod.yml --project-name tutor_local exec mongodb mongo --eval db.adminCommand({ setFeatureCompatibilityVersion: "4.0" })
We add a "sleep" statement to the upgrade process to ensure that the mongodb
container is available.
Turns out, the authentication mechanism should only be defined if there
is an actual authentication. For now, because of the urgency, we
hardcode this auth_mech to ":scram". We'll add a way to override it if
necessary, in the future.
- [Improvement] Make it easier to run edx-platform unit tests.
- [Bugfix] Fix segmentation fault during `tutor config save` on Mac OS M1 (#473). Thanks @ghassanmas!
- [Bugfix] Fix a bug that prevented connecting to external MongoDB instances.
- [Improvement] Make sure that the logo included in email notifications (including discussion responses) is the same as the site logo.
- [Bugfix] Install IPython directly from pypi instead of installing it from source (the reason it was installed from source is no longer relevant). The effect of this shall speed up the process of building the openedx-dev Docker image.
- [Feature] Add "openedx-dockerfile-post-git-checkout" patch.
- [Improvement] In the "openedx" Docker images, convert git patches to cherry-picks for a cleaner source tree.
- 💥[Feature] Make it possible to override local job configuration. This deprecates the older model for running jobs which dates back from a long time ago.
It should be unnecessary to build a custom openedx-dev Docker image. All tests
can run from within the dev Docker image, with a couple additional environment
variables.
Previously, the logo included in emails was loaded from edX' CDN. Here, we make
sure that the logo is actually the same as the site logo. Because the logo may
be theme-specific, we need to compute the logo url at runtime, and use a
lazily-evaluated string.
Close #447.
