This is a very large refactoring which aims at making Tutor both more
extendable and more generic. Historically, the Tutor plugin system was
designed as an ad-hoc solution to allow developers to modify their own
Open edX platforms without having to fork Tutor. The plugin API was
simple, but limited, because of its ad-hoc nature. As a consequence,
there were many things that plugin developers could not do, such as
extending different parts of the CLI or adding custom template filters.
Here, we refactor the whole codebase to make use of a generic plugin
system. This system was inspired by the Wordpress plugin API and the
Open edX "hooks and filters" API. The various components are added to a
small core thanks to a set of actions and filters. Actions are callback
functions that can be triggered at different points of the application
lifecycle. Filters are functions that modify some data. Both actions and
filters are collectively named as "hooks". Hooks can optionally be
created within a certain context, which makes it easier to keep track of
which application created which callback.
This new hooks system allows us to provide a Python API that developers
can use to extend their applications. The API reference is added to the
documentation, along with a new plugin development tutorial.
The plugin v0 API remains supported for backward compatibility of
existing plugins.
Done:
- Do not load commands from plugins which are not enabled.
- Load enabled plugins once on start.
- Implement contexts for actions and filters, which allow us to keep track of
the source of every hook.
- Migrate patches
- Migrate commands
- Migrate plugin detection
- Migrate templates_root
- Migrate config
- Migrate template environment globals and filters
- Migrate hooks to tasks
- Generate hook documentation
- Generate patch reference documentation
- Add the concept of action priority
Close #499.
Previously, the `k8s exec` command did not support unknown "--options". This
made it impossible to launch, say, a django shell in the lms container.
While implementing this feature we saw an opportunity to simplify the way jobs
are handled in the k8s commands.
Close #636.
Another related issue is: https://github.com/overhangio/2u-tutor-adoption/issues/52
- [Security] Apply SAML security fix.
- [Improvement] In addition to the Docker build arguments
`EDX_PLATFORM_REPOSITORY` and `NPM_REGISTRY`, also support two corresponding
and identically-named `config.yml` values serving the same purpose.
Previously, the only way for Tutor users to use a fork of edx-platform
or a custom NPM registry was to use build args during the image build.
This is suboptimal in the case of automatically building images from
CI pipelines, which may want to auto-detect when an image needs to be
rebuilt based on config.yml changes.
In addition, the EDX_PLATFORM_VERSION build argument can already be
set via a corresponding config.yml parameter (OPENEDX_COMMON_VERSION),
so it's reasonable to follow that precedent and also introduce
config.yml parameters to correspond with the EDX_PLATFORM_REPOSITORY
and NPM_REGISTRY build arguments.
Thus, introduce two new configuration parameters:
- EDX_PLATFORM_REPOSITORY
- NPM_REGISTRY
These parameters can now optionally be used instead of the
aforementioned build args.
- [Bugfix] Fix dockerize on arm64 by switching to the [powerman/dockerize](https://github.com/powerman/dockerize) fork (#591).
- [Bugfix] Fix "Unexpected args" error during service initialization on Kubernetes (#611).
The version of dockerize that shipped with the "openedx" image was not
compatible with arm64. The original project is unmaintained, but there
is a fork that provides a version that is compatible with arm64.
This was tested on arm64 with buildx:
docker buildx build --tag=openedx --platform=linux/arm64 ~/.local/share/tutor/env/build/openedx
Close #591
- [Bugfix] Fix `local/k8s quickstart` commands when upgrading from an older release (#595).
- [Bugfix] Fix running the default exim-relay SMTP server on arm64 (#600).
- [Feature] Add `tutor k8s apply` comand, which is a direct interface with `kubectl apply`.
- [Feature] Add `openedx-dockerfile-minimal` patch, which you can use to install custom packages and run commands as root in the Docker image.
- [Security] Fix vulnerability in call to invalid enrollment API (see [commit](e9369cffde)).
- [Bugfix] Fix "Internal Server Error / AttributeError / object has no attribute 'get_metadata'" in learning MFE.
- [Improvement] Replace all links to github.com/edx by github.com/openedx, following the migration of all repositories.
- [Bugfix] Fix `k8s start caddy` command.
- [Bugfix] Fix authentication in development due to missing SameSite policy on session ID cookie.
- [Bugfix] Display properly themed favicon.ico image in LMS, Studio and microfrontends.
- [Bugfix] Fix "LazyStaticAbsoluteUrl is not JSON serializable" error when sending bulk emails.
- [Bugfix] Fix `tutor local importdemocourse` fails when platform is not up.
In development, it was no longer possible to authenticate to the lms. Ater
signing in, the session ID could not be dropped, and thus the user was not
signed in, although no error was logged -- just a warning in the browser
console.
This problem was caused by the fact that the SameSite policy was set to "None"
in development.
Previously, we were redirecting all /*favicon.ico requests to the default
favicon. This meant that the favicon might not necessarily be correctly themed,
most notably in MFEs. Here, we resolve this issue by redirecting to the
theme-agnostic theming/asset/* url. Also, we restrict the overly generic regexp
for favicon url matching. We verified that we did not miss any url by running
the following command on the demo server:
tutor local logs caddy | grep --only-matching "host.*favicon.ico" | sort | uniq
The LazyStaticAbsoluteUrl object was breaking bulk emails again with the
following stacktrace:
2022-01-11 13:50:10,591 ERROR 12 [celery.app.trace] [user None] [ip None] trace.py:255 - Task lms.djangoapps.instructor_task.tasks.send_bulk_course_email[26b93357-018a-408f-b3f7-b69722447c5b] raised unexpected: EncodeError(TypeError('Object of type LazyStaticAbsoluteUrl is not JSON serializable'))
Traceback (most recent call last):
File "/openedx/venv/lib/python3.8/site-packages/kombu/serialization.py", line 50, in _reraise_errors
yield
File "/openedx/venv/lib/python3.8/site-packages/kombu/serialization.py", line 221, in dumps
payload = encoder(data)
File "/openedx/venv/lib/python3.8/site-packages/kombu/utils/json.py", line 69, in dumps
return _dumps(s, cls=cls or _default_encoder,
File "/openedx/venv/lib/python3.8/site-packages/simplejson/__init__.py", line 398, in dumps
return cls(
File "/openedx/venv/lib/python3.8/site-packages/simplejson/encoder.py", line 296, in encode
chunks = self.iterencode(o, _one_shot=True)
File "/openedx/venv/lib/python3.8/site-packages/simplejson/encoder.py", line 378, in iterencode
return _iterencode(o, 0)
File "/openedx/venv/lib/python3.8/site-packages/kombu/utils/json.py", line 59, in default
return super(JSONEncoder, self).default(o)
File "/openedx/venv/lib/python3.8/site-packages/simplejson/encoder.py", line 272, in default
raise TypeError('Object of type %s is not JSON serializable' %
TypeError: Object of type LazyStaticAbsoluteUrl is not JSON serializable
The point of that lazy object was to link to the lms logo even when a custom
theme was enabled. Luckily, we no longer need this lazy evaluation because we
now have theme-agnostic urls that point to static asset (see
https://github.com/openedx/edx-platform/pull/29461).
See:
https://discuss.overhang.io/t/error-while-sending-bulk-emails-lazystaticabsoluteurl-is-not-json-serializable/2176/
upgrading.
- [Bugfix] During upgrade, make sure that environment is up-to-date
prior to prompting to rebuild the custom images.
- [Bugfix] Fix ownership of mysql data, in particular when upgrading a
Kubernetes cluster to Maple.
- [Bugfix] Ensure that ``tutor k8s upgrade`` is run during ``tutor k8s
quickstart``, when necessary.
- 💥[Bugfix] By default, detect the current version during ``tutor
k8s/local upgrade``.
- [Bugfix] Fix upgrading from Lilac to Maple on Kubernetes by deleting
deployments and services.
`upgrade` had several issues, which are summarized here:
https://discuss.overhang.io/t/confusing-instructions-during-upgrade/2281/7
- The docs say that you should run quickstart, but what most people will see is
the big command tutor local upgrade --from=lilac verbatim paragraph.
- The local upgrade command should be very explicit about the fact that users
need to run quickstart.
- Maybe the name of the local upgrade command should be improved.
- When upgrading tutor from one major release to the next, there should be a
more explicit warning to inform users of what they are doing (see this other
conversation 1)
- We should tell people that they almost certainly need to enable the tutor and
the mfe plugins, if they are not enabled during upgrade.
- A link to all of the breaking changes from the changelog should be
prominently displayed during upgrade.
- The docs should emphasize that upgrading from one major release to the next
is potentially a risky endeavor and that downgrading is not possible. The docs
should also link to the changelog.
This commit has grown slightly beyond the intended scope, but the changes should be mostly positive.
In theory, we can assign ownership of mysql data to just any user. But in
Lilac, mysql was running with user 999. When upgrading to Maple, on Kubernetes,
the fsGroupChangePolicy was causing a change of the data *group* (to 1000) but
not of the user. This was causing a crash with the following error:
[ERROR] InnoDB: The error means mysqld does not have the access rights to the directory.
When upgrading from Lilac, all services break with the following error:
Service "***" is invalid: spec.ports[0].nodePort: Forbidden: may not be used when `type` is 'ClusterIP'
Upgrading deployments fails as well:
Deployment.apps "***" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{"app.kubernetes.io/instance":"openedx-********", "app.kubernetes.io/managed-by":"tutor", "app.kubernetes.io/name":"***", "app.kubernetes.io/part-of":"openedx"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable
That's because deployments and services need to be deleted as part of the Maple
upgrade. So that's what we do as part of `tutor k8s upgrade --from=lilac`. And
we take the opportunity to:
1. Run upgrade as part of quickstart, when necessary.
2. Default to lilac during `tutor k8s upgrade`.
Close #551.
- [Security] Upgrade Django to 3.2.11 in edx-platform.
- [Security] Prevent non-staff users from searching usernames by email by
abusing the logout url.
Without this patch, it is possible to search for account info including
username by using the email of a learner. This fix disallows searching using
email by regular users and restricts this feature to only staff and superusers.
- A shared cookie domain between lms and cms is no longer recommended:
https://github.com/edx/edx-platform/blob/master/docs/guides/studio_oauth.rst
- refactor: clean mounted data folder in lms/cms. In Lilac, the
bind-mounted lms/data and cms/data folders are a mess because new
folders are created there for every new course organisation. These
folders are empty. As far as we know they are useless... With this
change we move these folders to a dedicated "modulestore" subdirectory;
which corresponds better to the initial intent of the fs_root setting.
- fix: frontend failure during login to the lms. See:
https://github.com/openedx/build-test-release-wg/issues/104
- feat: move all forum-related code to a dedicated plugin. Forum is an
optional feature, and as such it deserves its own plugin. Starting from
Maple, users will be able to install the forum from
https://github.com/overhangio/tutor-forum/
- migrate from DCS_* session cookie settings to SESSION_*. That's
because edx-platform no longer depends on django-cookies-samesite. Close
https://github.com/openedx/build-test-release-wg/issues/110
- get rid of tons of deprecation warnings in the lms/cms
- feat: make it possible to point to themed assets. Cherry-picking this
change makes it possible to point to themed assets with a theme-agnostic
url, notably from MFEs.
- Install all official plugins as part of the `tutor[full]` package.
- Don't print error messages about loading plugins during autocompletion.
- Prompt for image building when upgrading from one release to the next.
- Add `tutor local start --skip-build` option to skip building Docker images.
Close #450.
Close #545.
Régis Behmo
Kyle McCormick
Florian Haas
Rebecca Graber
Emad Rad
iamcristye@outlook.com