Bumping the `OPENEDX_COMMON_VERSION` in the master branch usually
creates a conflict when we merge the change in the nightly branch. To
avoid this conflict, we add some logic to the `OPENEDX_COMMON_VERSION`.
This change should be invisible for most users.
This partially addresses issue #936.
MySQL 8 defaults to a binlog expiry period of 2592000 seconds
(30 days), which for Tutor/Open edX purposes can be considered
excessive.
On the one hand, it is unlikely that a MySQL server configured for
Tutor uses MySQL replication at all (considering that up until Tutor
15 and MySQL 5.7, the binlog was disabled by default, rendering
replication impossible). Even if it does, a replica lagging more than
two days behind the primary server would be unacceptable.
Likewise, it is unlikely that an Open edX database is backed up less
than once a day, thus is is unlikely that Open edX admins would
benefit from the ability to do point-in-time restore over a 30-day
period.
On the other hand, having a 30-day binlog expiry period can
considerably increase the storage space requirements for the MySQL
container, particularly on busy Open edX platforms. When left
unchecked, this can even cause the MySQL container to run into "No
space left on device" situations, disabling the MySQL database
altogether. Thus, the MySQL default settings are likely to be a net
disadvantage for Open edX admins.
Finally, all of the above considerations apply only if the Open edX
administrator has chosen to run their own MySQL and not opted for a
DBaaS solution like AWS RDS.
Thus, it should be acceptable to run with a reduced binlog expiry
period of 3 days (rather than 30) by default.
Therefore, inject the --binlog-expire-logs-seconds=259200 argument
into the Tutor-generated command to start mysqld.
Reference:
https://dev.mysql.com/doc/refman/8.0/en/replication-options-binary-log.html#sysvar_binlog_expire_logs_seconds
On macOS, building the "openedx-dev" Docker image resulted in an image that required more than 600 GB of disk space. This was due to the `adduser` command which was called with a user ID of 2x10⁹ (on macOS only). This resulted in a very large /var/log/faillog file, hence the image size.
Related upstream discussion: https://github.com/moby/moby/issues/5419
Close https://github.com/openedx/wg-developer-experience/issues/178
On Oct. 10, py2neo package was abruptly removed from pypi, GitHub, and
the py2neo website now displays just a super funny meme: https://py2neo.org/
Yes, we should get rid of that dependency, but we are still supposed to
support existing users. So we install py2neo from our fork.
The LMS was overriding CMS_BASE properly, but Studio (CMS) configuration
was not. That meant that Studio's CMS_BASE in dev mode was using the
devstack default of localhost:18010 (because this is what's defined in
edx-platform). This in turn broke parts of Studio that use this value,
such as the XBlock v2 API (/api/xblock/v2).
This commit derives the value of the CMS_BASE Django setting from
Tutor's CMS_HOST config value, in the same way that the LMS does it.
Pound keys were interpreted as comments. This is annoying when we want
to parse html color codes, such as in:
$ tutor config save --set "INDIGO_PRIMARY_COLOR=#225522"
$ tutor config printvalue INDIGO_PRIMARY_COLOR
null
Close #866.
This is a follow-up fix to #819, where the corresponding change was
added to the mysqld invocation in the "tutor local" (that is,
docker-compose) deployment method, but omitted from its "tutor k8s"
equivalent.
This fix is for a rather serious issue that affects users who upgrade
from Olive to Palm. The client mysql charset and collation was
incorrectly set to utf8mb4, while the server stil runs utf8mb3. Only
users who run the mysql container are affected.
To resolve this issue, we explicitely configure the client to use the
utf8mb3 charset/collation.
Important note: users who have somehow managed to upgrade from olive to
Palm before may find themselves in an undefined state. They might have
to fix their mysql data manually. Same thing for users who launched Palm
from scratch; although, according to my preliinary tests, they should be
able to downgrade their connection from utf8mb4 to utf8mb3 without
issue.
In addition, we upgrade to mysql 8.1.0. Among many other fixes, this
avoids a server restart after the upgrade:
> An in-place upgrade from MySQL 5.7 to MySQL 8.0, without a server
> restart, could result in unexpected errors when executing queries on
> tables. This fix eliminates the need to restart the server between the
> upgrade and queries. (Bug #35410528)
https://dev.mysql.com/doc/relnotes/mysql/8.0/en/news-8-0-34.html
See also the 8.1.0 release notes:
https://dev.mysql.com/doc/relnotes/mysql/8.1/en/news-8-1-0.html
Close #887.
`tutor ... do settheme default` is meant to revert to the
default theme. However, in its current implementation, it
creates SiteTheme objects pointing to a theme named "default",
which doesn't exist, resulting in errors like:
Theme dirs:
[Path('/openedx/themes')]]
Traceback (most recent call last):
File "/openedx/edx-platform/openedx/core/djangoapps/theming/helpers.py", line 204, in get_current_theme
themes_base_dir=get_theme_base_dir(site_theme.theme_dir_name),
File "/openedx/edx-platform/openedx/core/djangoapps/theming/helpers.py", line 242, in get_theme_base_dir
raise ValueError(
ValueError: Theme 'default' not found in any of the following themes dirs,
This works from the perspective of the user, because a missing theme is
treated as the default theme. However, the errors are unneccesary &
confusing.
By simply deleting & not recreating SiteTheme objects instead,
we are able to revert to the default theme while keeping the
logs clear of theming errors.
Manual configuration via the `MOUNTS` setting was inconvenient. We
(re)introduce a new(ish) `tutor mounts` command. Old timers will perhaps
remember that we used to have a `tutor bindmount` command. Well, it's
back! But better and different.
Templated hooks we almost completely useless, so we get rid of them.
This allows us to get rid entirely of hook names and hook indexes, which
makes the whole implementation much simpler. Hook removal (with
`clear_all`) is achieved thanks to weak references.
This is an important change, where we get remove the previous `--mount`
option, and instead opt for persistent bind-mounts.
Persistent bind mounts have several advantages:
- They make it easier to remember which folders need to be bind-mounted.
- Code is *much* less clunky, as we no longer need to generate temporary
docker-compose files.
- They allow us to bind-mount host directories *at build time* using the
buildx `--build-context` option.
- The transition from development to production becomes much easier, as
images will automatically be built using the host repo.
The only drawback is that persistent bind-mounts are slightly less
portable: when a config.yml file is moved to a different folder, many
things will break if the repo is not checked out in the same path.
For instance, this is how to start working on a local fork of
edx-platform:
tutor config save --append MOUNTS=/path/to/edx-platform
And that's all there is to it. No, this fork will be used whenever we
run:
tutor images build openedx
tutor local start
tutor dev start
This change is made possible by huge improvements in the build time
performance. These improvements make it convenient to re-build Docker
images often.
Related issues:
https://github.com/openedx/wg-developer-experience/issues/71https://github.com/openedx/wg-developer-experience/issues/66https://github.com/openedx/wg-developer-experience/issues/166
It was useless to create a *-permissions job for every application.
Instead, we create a single "permissions" service. It can be extended
via the "docker-compose-permissions-command" patch.
Instead, the compose plugin must be installed.
We deprecate docker-compose because v1 is not supported starting from
the end of June 2023.
See "evolution of compose": https://docs.docker.com/compose/compose-v2/
Among other changes: ORA2 file uploads were stored in a folder named
"SET-ME-PLEASE (ex. bucket-name)" (sigh). With this change, the folder
should be automatically renamed to "openedxuploads". This issue has been
occuring since June 2019... (sigh²)
Close #707
See:
https://github.com/overhangio/tutor/actions/runs/5260213022/jobs/9506811909#step:9:33
`scriv github-release --repo=overhangio/tutor` causes the following
error:
Traceback (most recent call last):
File "/opt/hostedtoolcache/Python/3.7.16/x64/bin/scriv", line 8, in
<module>
sys.exit(cli())
File
"/opt/hostedtoolcache/Python/3.7.16/x64/lib/python3.7/site-packages/click/core.py",
line 1130, in __call__
return self.main(*args, **kwargs)
File
"/opt/hostedtoolcache/Python/3.7.16/x64/lib/python3.7/site-packages/click/core.py",
line 1055, in main
rv = self.invoke(ctx)
File
"/opt/hostedtoolcache/Python/3.7.16/x64/lib/python3.7/site-packages/click/core.py",
line 1657, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File
"/opt/hostedtoolcache/Python/3.7.16/x64/lib/python3.7/site-packages/click/core.py",
line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
File
"/opt/hostedtoolcache/Python/3.7.16/x64/lib/python3.7/site-packages/click/core.py",
line 760, in invoke
return __callback(*args, **kwargs)
File
"/opt/hostedtoolcache/Python/3.7.16/x64/lib/python3.7/site-packages/scriv/ghrel.py",
line 99, in github_release
config=scriv.config,
File
"/opt/hostedtoolcache/Python/3.7.16/x64/lib/python3.7/site-packages/jinja2/environment.py",
line 1301, in render
self.environment.handle_exception()
File
"/opt/hostedtoolcache/Python/3.7.16/x64/lib/python3.7/site-packages/jinja2/environment.py",
line 936, in handle_exception
raise rewrite_traceback_stack(source=source)
File "<template>", line 3, in top-level template code
TypeError: 'Version' object is not subscriptable
This paves the way for `docker buildx build` and better caching.
For instance, with this change you can try out the following plugin,
which should make image building much faster in CI:
https://gist.github.com/regisb/4049622ec4b48cbd48c89ec708dc5252
(not ready for production just yet, we still need to build and push the
images)
In the output of `plugins list` it's difficult to see which plugins are
enabled at a glance. This change adds a more visible checkmark to the
output.
Ex:
$ tutor plugins list
NAME STATUS VERSION
cairn ✅ enabled 15.0.3
discovery installed 15.0.0
ecommerce installed 15.0.1
forum installed 14.0.0
mfe ✅ enabled 15.0.5
minio installed 15.1.0
Before this commit, setting up an edx-platform development environment
took multiple steps:
tutor dev launch
tutor dev run --mount=/path/to/edx-platform lms bash
>> pip install -e .
>> npm clean-install
>> openedx-assets build --env=dev
This commit moves the steps under ``run`` into an init task, which
is automatically run by ``launch``. Thus, setup is now one command:
tutor dev launch --mount=edx-platform
These extra init steps are only applicable when bind-mounting
edx-platform (because bind-mounting the repository overrides
some important artifacts that exist on the image, which must be
re-generated). Thus, the new init tasks exists early if it detects
that it is *not* operating on a bind-mounted repository.
Finally, we try to simplify the Open edX development docs so that
it is clearer how bind-mounting fits into the development process.
These bind-mounts:
* ../build/openedx/themes:/openedx/themes
* ../build/openedx/requirements:/openedx/requirements
existed in the dev lms and cms containers, but they did
not exist in the lms-job and cms-job containers.
This means that themes and requirements that were *built into the
image* would exist in the job containers, but live updates to the
themes and requirements would not apply.
To resolve this, we set ``volumes:`` on the lms-job and cms-job
services so that they match the volumes for the normal lms and
cms services.
Part of: https://github.com/openedx/wg-developer-experience/issues/146
Closes: https://github.com/openedx/wg-developer-experience/issues/152
This works around (but does not close) these related issues:
* https://github.com/openedx/wg-developer-experience/issues/150
* https://github.com/openedx/wg-developer-experience/issues/151
The lms-job and cms-job services were configured to use
{{ DOCKER_IMAGE_OPENEDX }} rather than {{ DOCKER_IMAGE_OPENEDX_DEV }}.
This means that when running jobs in dev mode, a la:
tutor dev do init
a production image would be used, to the user's surprise.
When a job is invoked, we now replace the job in k8s/jobs.yml
instead of rewriting jobs.yml to only contain the relevant
job. This allows patchStrategicMerge to work for jobs.
