6
0
mirror of https://github.com/ChristianLight/tutor.git synced 2024-12-12 22:27:47 +00:00
Commit Graph

629 Commits

Author SHA1 Message Date
Régis Behmo
c4388e134c v14.0.0: upgrade to Nutmeg
- 💥 [Feature] Upgrade to Nutmeg: (by @regisb)
    - 💥 [Feature] Persistent grades are now enabled by default.
    - [Bugfix] Remove edX references from bulk emails ([issue](https://github.com/openedx/build-test-release-wg/issues/100)).
    - [Improvement] For Tutor Nightly (and only Nightly), official plugins are now installed from their nightly branches on GitHub instead of a version range on PyPI. This will allow Nightly users to install all official plugins by running ``pip install -e ".[full]"``.
    - [Bugfix] Start MongoDB when running migrations, because a new data migration fails if MongoDB is not running
2022-06-09 19:18:20 +02:00
Régis Behmo
698da4b1ad v13.3.1 (2022-06-06)
- [Fix] Crashing celery workers in development (#681). (by @regisb)
- [Fix] Fix studio logout issue. (by @Alec4r)
2022-06-06 16:28:47 +02:00
Régis Behmo
74212da9e2 fix: celery workers in dev
Celery workers failed to start in development with the following stacktrace:

    cms-worker_1                 | Traceback (most recent call last):
    cms-worker_1                 |   File "/openedx/venv/bin/celery", line 8, in <module>
    cms-worker_1                 |     sys.exit(main())
    cms-worker_1                 |   File "/openedx/venv/lib/python3.8/site-packages/celery/__main__.py", line 16, in main
    cms-worker_1                 |     _main()
    cms-worker_1                 |   File "/openedx/venv/lib/python3.8/site-packages/celery/bin/celery.py", line 322, in main
    cms-worker_1                 |     cmd.execute_from_commandline(argv)
    cms-worker_1                 |   File "/openedx/venv/lib/python3.8/site-packages/celery/bin/celery.py", line 499, in execute_from_commandline
    cms-worker_1                 |     super(CeleryCommand, self).execute_from_commandline(argv)))
    cms-worker_1                 |   File "/openedx/venv/lib/python3.8/site-packages/celery/bin/base.py", line 305, in execute_from_commandline
    cms-worker_1                 |     return self.handle_argv(self.prog_name, argv[1:])
    cms-worker_1                 |   File "/openedx/venv/lib/python3.8/site-packages/celery/bin/celery.py", line 491, in handle_argv
    cms-worker_1                 |     return self.execute(command, argv)
    cms-worker_1                 |   File "/openedx/venv/lib/python3.8/site-packages/celery/bin/celery.py", line 415, in execute
    cms-worker_1                 |     return cls(
    cms-worker_1                 |   File "/openedx/venv/lib/python3.8/site-packages/celery/bin/worker.py", line 221, in run_from_argv
    cms-worker_1                 |     *self.parse_options(prog_name, argv, command))
    cms-worker_1                 |   File "/openedx/venv/lib/python3.8/site-packages/celery/bin/base.py", line 428, in parse_options
    cms-worker_1                 |     self.parser = self.create_parser(prog_name, command)
    cms-worker_1                 |   File "/openedx/venv/lib/python3.8/site-packages/celery/bin/base.py", line 440, in create_parser
    cms-worker_1                 |     description=self._format_description(self.description),
    cms-worker_1                 |   File "/openedx/venv/lib/python3.8/site-packages/celery/bin/base.py", line 462, in _format_description
    cms-worker_1                 |     text.fill_paragraphs(text.dedent(description), width))
    cms-worker_1                 |   File "/openedx/venv/lib/python3.8/site-packages/celery/utils/text.py", line 58, in fill_paragraphs
    cms-worker_1                 |     return sep.join(fill(p, width) for p in s.split(sep))
    cms-worker_1                 |   File "/openedx/venv/lib/python3.8/site-packages/celery/utils/text.py", line 58, in <genexpr>
    cms-worker_1                 |     return sep.join(fill(p, width) for p in s.split(sep))
    cms-worker_1                 |   File "/opt/pyenv/versions/3.8.12/lib/python3.8/textwrap.py", line 391, in fill
    cms-worker_1                 |     return w.fill(text)
    cms-worker_1                 |   File "/opt/pyenv/versions/3.8.12/lib/python3.8/textwrap.py", line 363, in fill
    cms-worker_1                 |     return "\n".join(self.wrap(text))
    cms-worker_1                 |   File "/opt/pyenv/versions/3.8.12/lib/python3.8/textwrap.py", line 354, in wrap
    cms-worker_1                 |     return self._wrap_chunks(chunks)
    cms-worker_1                 |   File "/opt/pyenv/versions/3.8.12/lib/python3.8/textwrap.py", line 248, in _wrap_chunks
    cms-worker_1                 |     raise ValueError("invalid width %r (must be > 0)" % self.width)
    cms-worker_1                 | ValueError: invalid width -2 (must be > 0)

This issue was reported upstream here: https://github.com/celery/celery/issues/6302

It is caused by the `tty: true` statement, for some reason. It will be fixed in
Nutmeg, after celery is upgraded to 5.2.6.

Close #681.
2022-06-06 16:25:35 +02:00
Alejandro Cardenas
f6b7826520 fix: change studio url in IDA_LOGOUT_URI_LIST
fix: remove FRONTEND_LOGOUT_URL to use the default

docs: update CHANGELOG
2022-06-06 16:13:23 +02:00
Régis Behmo
f9b4228033 v13.3.0 (2022-06-03)
- [Security] Apply logout redirect url security fix. (by @regisb)
- [Feature] Make it possible to force the rendering of a given template, even when the template path matches an ignore pattern. (by @regisb)
- 💥[Fix] Get rid of the `tutor config render` command, which is useless now that themes can be implemented as plugins. (by @regisb)
2022-06-03 15:00:03 +02:00
Régis Behmo
79eca380f6 security: apply logout redirect security patch 2022-06-03 14:58:32 +02:00
Régis Behmo
82f2a448d2 feat: render files in ignored directories
When rendering theme files in a plugin, the *.scss files are stored in a
"partials" subdirectory, which was ignored by the environment rendering logic.
To render these files, we move the path ignoring logic to a filter, which is a
list of regular expressions. Values in this filter can be overridden by another
filter.

See the corresponding issue in the indigo theme plugin:
https://github.com/overhangio/tutor-indigo/issues/24
2022-06-03 12:54:27 +02:00
Régis Behmo
1e0c305508 fix: get rid of the tutor config render command
This command is useless now that we can implement themes as plugins. This
allows us to considerably simplify the Renderer class constructor.
2022-06-03 12:54:27 +02:00
Régis Behmo
669f836328 v13.2.3 (2022-05-30)
- [Fix] Truncate site display name to 50 characters with a warning, fixing data too long error for long site names. (by @navinkarkera)
- [Feature] Add patch to allow overriding final openedx docker image CMD.
- [Fix] Ignore Python plugins that cannot be loaded. (by @regisb)
- [Improvement] Faster and more reliable builds with `npm clean-install` instead of `npm install`. (by @regisb. Thanks @ghassanmas!)
- [Fix] Fix 500 error during studio login. (by @regisb)
- [Fix] Fix updates for the Caddy deployment in multi-node Kubernetes clusters (#660). Previously, Caddy configuration updates might fail if the Kubernetes cluster had more than one worker node. (by @fghaas)
2022-05-30 14:33:32 +02:00
Navin Karkera
3890a38ca1 fix: truncate site name if longer than 50 characters and show warning
Close #518
2022-05-30 12:51:11 +02:00
Daniel Rudnitski
bae65ff701 feat: patch for openedx-dockerfile-final 2022-05-30 10:34:53 +02:00
Régis Behmo
6fb0a6b855 fix: ignore plugins that cannot be loaded
When running multiple concurrent versions of a plugin there are sometimes
version conflicts that prevent the plugin from being loaded. Prior to v1, Tutor
was correctly ignoring plugins that could not be loaded. During the transition
to v1 we lost that feature because we only captured TutorErrors.
2022-05-24 11:38:37 +02:00
Régis Behmo
4bbeb4b84f feat: pinned nodejs requirements with npm ci
Contrary to what we might expect, `npm install` does not install pinned
requirements from a project's package-lock.json. That's the responsibility of
`npm ci`: https://docs.npmjs.com/cli/v8/commands/npm-ci

Running `npm ci` is also *much* faster than `npm install`, so that's a huge win.

See this issue for reference: https://github.com/openedx/frontend-wg/issues/100
2022-05-24 11:17:14 +02:00
Florian Haas
78424776b6 fix: Enable rolling updates for the Caddy deployment in multi-node Kubernetes
When a Pod associated with a Deployment is updated (for example, due
to a change to its ConfigMap, or an updated image reference),
Kubernetes uses a ReplicaSet to spin up a Pod with the new
configuration, and once it is up, it tears down the old one.

In case of the Caddy Deployment, this is complicated by the fact that
it uses a Persistent Volume Claim (PVC), whose corresponding volume
uses a Read/Write-Once (RWO) configuration. This means that it can
only be used by multiple Pods if all those Pods all run on the same
Kubernetes worker node.

In order to enable rolling upgrades for the Caddy Deployment, we need
to ensure that its replacement Pod is scheduled on the same node as
the original Pod.

Thus, add a pod affinity rule that will force exactly that behavior.

Reference:
https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/

The other Tutor services that use volumes (MySQL, Redis, Elasticsearch
and MongoDB) do not need this fix, since they all use the "Recreate"
deployment strategy: their Pods are all automatically torn down before
being replaced. This strategy is not needed for Caddy, and using a pod
affinity rule is less disruptive to the learner experience.
2022-05-19 12:51:23 +02:00
Régis Behmo
549922f0b9 fix: 500 error during studio login
Login to the studio was causing a 500 error, although the user was correctly
connected. See the issue description here:
https://github.com/openedx/build-test-release-wg/issues/114

The error was caused by the fact that both LMS and CMS session cookies were
being sent to the CMS and the LMS session cookie was considered the "right"
one. To clarify between both cookies, we need to have different session cookie
names between the lms and the cms. This requirement was actually documented in
step 5 of this authentication migration document:
https://github.com/openedx/edx-platform/blob/master/docs/guides/studio_oauth.rst#migration

For more information, have a look at the live stream during which this issue
was debugged: https://youtu.be/m3t1U_y6F64

Close https://github.com/openedx/build-test-release-wg/issues/114
2022-05-19 09:55:28 +02:00
Kyle McCormick
b55df07eba v13.2.2 2022-05-07 13:37:57 +02:00
Kyle McCormick
5d08438a7b docs: attribute @kdmccormick's changes since v13.2.0 2022-05-07 13:37:57 +02:00
Régis Behmo
4b587e41d4 - [Fix] Fix broken file upload in studio because of unpinned studio-frontend requirement (see [discussion](https://discuss.overhang.io/t/missing-js-css-files-missing-from-openedx-docker-image-in-studio/2629) and [pull request](https://github.com/openedx/edx-platform/pull/30309)) (by @regisb. Thanks @uetuluk!).
- [Fix] "The Compose file is invalid" error on mounting dev-only
  folders. (by @regisb)
- [Fix] CMS settings in development. (by @regisb)
2022-05-06 15:01:47 +02:00
Régis Behmo
85a44213b7 docs: improve changelog to highlight 3rd party contributions
With this change, we want to better highlight the contributions of
developers to Tutor. We want to publicly acknowledge the positive impact
that individuals and companies have on the development of the platform.
to that end, each changelog entry can now be suffixed with the name of
the author (individual or company) who authored the change. These names
will find their way to the release notes for every release. Eventually,
we also want to spread these release notes more widely. For instance, we
could post new releases to the forum to notify the community of
important changes.

If you have contributed to Tutor in the past, feel free to open a PR and
append your name to the changes that you made. We will not be able to
update the release notes for every release out there, but your
contributions will be acknowledged from the changelog.
2022-05-06 13:58:59 +01:00
Régis Behmo
3290def1ef fix: broken file upload in studio
see discussion: https://discuss.overhang.io/t/missing-js-css-files-missing-from-openedx-docker-image-in-studio/2629
see pull request: https://github.com/openedx/edx-platform/pull/30309
(thanks @uetuluk!)
2022-05-06 13:51:07 +01:00
Régis Behmo
514e3fce22 fix: compose error on "dev --mount=learning:..."
When mounting a directory in a dev-only container, such as the
"learning" mfe, docker-compose is failing because it is attempting to
run "docker-compose stop" in the local context -- which knows nothing
about the learning container.

To resolve this, we store tmp volumes either in the local or dev
docker-compose.yml, and load either one depending on the context.
2022-04-26 14:11:26 +01:00
Régis Behmo
75ed5b4609 fix: cms settings in development
See:
https://discuss.overhang.io/t/lms-and-cms-containers-both-run-lms-in-v13-2-0-dev/2693/2
2022-04-25 12:50:05 +02:00
Régis Behmo
db455a1379 v13.2.0 (2022-04-24)
- [Improvement] Add the `COMPOSE_PROJECT_STARTED` action and run `dev
  stop` on `local start` (and vice versa).
- [Feature] Introduce `local/dev copyfrom` command to copy contents from
  a container.
- [Bugfix] Fix a race condition that could prevent a newly provisioned
  LMS container from starting due to a `FileExistsError` when creating
data folders.
- [Deprecation] Mark `tutor dev runserver` as deprecated in favor of
  `tutor dev start`. Since `start` now supports bind-mounting and
breakpoint debugging, `runserver` is redundant and will be removed in a
future release.
- [Improvement] Allow breakpoint debugging when attached to a service
  via `tutor dev start SERVICE`.
- [Security] Apply rate limiting security fix (see
  [commit](b5723e416e)).
- [Feature] Introduce the ``-m/--mount`` option in ``local`` and ``dev``
  commands to auto-magically bind-mount folders from the host.
- [Feature] Add `tutor dev quickstart` command, which is similar to
  `tutor local quickstart`, except that it uses dev containers instead
of local production ones and includes some other small differences for
the convience of Open edX developers. This should remove some friction
from the Open edX development setup process, which previously required
that users provision using local producation containers (`tutor local
quickstart`) but then stop them and switch to dev containers (`tutor
local stop && tutor dev start -d`).
- 💥[Improvement] Make it possible to run `tutor k8s exec <command with
  multiple arguments>` (#636). As a consequence, it is no longer
possible to run quoted commands: `tutor k8s exec "<some command>"`.
Instead, you should remove the quotes: `tutor k8s exec <some command>`.
- 💥[Deprecation] Drop support for the `TUTOR_EDX_PLATFORM_SETTINGS`
  environment variable. It is now recommended to create a plugin
instead.
- 💥[Improvement] Complete overhaul of the plugin extension mechanism.
  Tutor now has a hook-based Python API: actions can be triggered at
different points of the application life cycle and data can be modified
thanks to custom filters. The v0 plugin API is still supported, for
backward compatibility, but plugin developers are encouraged to migrate
their plugins to the new API. See the new plugin tutorial for more
information.
- [Improvement] Improved the output of `tutor plugins list`.
- [Feature] Add `tutor [dev|local|k8s] status` command, which provides
  basic information about the platform's status.
2022-04-24 15:09:16 +02:00
Régis Behmo
7d20329894 feat: local stop on dev start (and vice versa)
Running `local start` while a dev platform is still running is a common sourse
of mistakes. Here we introduce a new action to automatically stop local and dev
projects whenever a project with a different name is started.
2022-04-24 09:56:12 +02:00
Régis Behmo
27449f4068 feat: add dev/local copyfrom commands
`copyfrom` copies data from a container to the local filesystem. It's similar
to bindmount, but less clunky, and more intuitive. Also, it plays along great
with `--mount`. Eventually we'll just get rid of the `bindmount` command and
the `--volume` option.
2022-04-24 09:51:46 +02:00
Braden MacDonald
fde20f0e8a fix: race condition could cause mkdirs() to fail with "dir exists" 2022-04-23 18:11:32 +02:00
Kyle McCormick
5a81b6a6cb feat: deprecate runserver in favor of start
`tutor dev runserver` will be removed in a future release.
Developers are encouraged to use `tutor dev start` instead,
which is more flexible and provides a consistent interface
with `tutor local start`.

As part of this deprecation, we enable the `tty` and
`stdin_open` options on development docker-compose
services. This will allow developers to use `start`
for breakpoint debugging, which was previously only
availble via `runserver`. Several parallel PRs have
been merged in order to make the same change in the
development services of the official plugins.

Although `start` does not support the `--volume` option,
it supports a more-powerful `--mount` option. So, where
developers previously used:

    tutor dev runserver --volume ...

to bind-mount host directories, they should now use:

    tutor dev start --mount ...

Resolves https://github.com/overhangio/2u-tutor-adoption/issues/61
2022-04-21 15:00:54 +02:00
Régis Behmo
658d6957db sec: apply rate limiting security fix 2022-04-20 19:48:34 +02:00
Régis Behmo
d9486018a2 feat: add --mount option to local/dev
The `--mount` option is available both with `tutor local`
and `tutor dev` commands. It allows users to easily bind-mount containers from
the host to containers. Yes, I know, we already provide that possibility with
the `bindmount` command and the `--volume=/path/` option. But these suffer from
the following drawbacks:

- They are difficult to understand.
- The "bindmount" command name does not make much sense.
- It's not convenient to mount an arbitrary folder from the host to multiple
  containers, such as the many lms/cms containers (web apps, celery workers and
  job runners).

To address this situation, we now recommend to make use of --mount:

1. `--mount=service1[,service2,...]:/host/path:/container/path`: manually mount
   `/host/path` to `/container/path` in container "service1" (and "service2").
2. `--mount=/host/path`: use the new v1 plugin API to discover plugins that
   will detect this option and select the right containers in which to bind-mount
   volumes. This is really nifty...

Close https://github.com/overhangio/2u-tutor-adoption/issues/43
2022-04-20 19:33:17 +02:00
Kyle McCormick
df0e26c58e feat: introduce tutor dev quickstart
Add `tutor dev quickstart` command, which is equivalent to
`tutor local quickstart`, but uses dev containers instead
of local production ones and includes some other small
differences for the convience of Open edX developers.
This should remove some friction
from the Open edX development setup process, which previously
required that users provision using local producation
containers but then stop them and switch to dev containers:
 * tutor local quickstart
 * tutor local stop
 * tutor dev start -d

Document the command and its improved workflow in
./docs/tutorials/nightly.rst

Fixes overhangio/2u-tutor-adoption#58
2022-04-19 16:53:57 +02:00
Régis Behmo
d5a790d5d0 refactor: get rid of the openedx Docker entrypoint
The entrypoint in the "openedx" Docker image was used only to define the
DJANGO_SETTINGS_MODULE environment variable, based on SERVICE_VARIANT and
SETTINGS. We ditch SETTINGS in favour of defining explicitely
DJANGO_SETTINGS_MODULE.

The problem with the Docker entrypoint is that it was bypassed whenever we ran
`tutor local exec` or `tutor k8s exec`. By removing it we make it simpler for
end-users to run manage.py commands in kubernetes.
2022-04-15 15:37:56 +02:00
Régis Behmo
15b219e235 feat: migrate to plugins.v1 with filters & actions
This is a very large refactoring which aims at making Tutor both more
extendable and more generic. Historically, the Tutor plugin system was
designed as an ad-hoc solution to allow developers to modify their own
Open edX platforms without having to fork Tutor. The plugin API was
simple, but limited, because of its ad-hoc nature. As a consequence,
there were many things that plugin developers could not do, such as
extending different parts of the CLI or adding custom template filters.

Here, we refactor the whole codebase to make use of a generic plugin
system. This system was inspired by the Wordpress plugin API and the
Open edX "hooks and filters" API. The various components are added to a
small core thanks to a set of actions and filters. Actions are callback
functions that can be triggered at different points of the application
lifecycle. Filters are functions that modify some data. Both actions and
filters are collectively named as "hooks". Hooks can optionally be
created within a certain context, which makes it easier to keep track of
which application created which callback.

This new hooks system allows us to provide a Python API that developers
can use to extend their applications. The API reference is added to the
documentation, along with a new plugin development tutorial.

The plugin v0 API remains supported for backward compatibility of
existing plugins.

Done:
- Do not load commands from plugins which are not enabled.
- Load enabled plugins once on start.
- Implement contexts for actions and filters, which allow us to keep track of
  the source of every hook.
- Migrate patches
- Migrate commands
- Migrate plugin detection
- Migrate templates_root
- Migrate config
- Migrate template environment globals and filters
- Migrate hooks to tasks
- Generate hook documentation
- Generate patch reference documentation
- Add the concept of action priority

Close #499.
2022-04-15 15:30:54 +02:00
Kyle McCormick
b6999824a7 feat: add tutor [dev|local|k8s] status command 2022-04-15 09:29:02 +02:00
Régis Behmo
cac9d14e01 feat: run tutor k8s exec ... commands with multiple args
Previously, the `k8s exec` command did not support unknown "--options". This
made it impossible to launch, say, a django shell in the lms container.

While implementing this feature we saw an opportunity to simplify the way jobs
are handled in the k8s commands.

Close #636.
Another related issue is: https://github.com/overhangio/2u-tutor-adoption/issues/52
2022-04-15 09:21:36 +02:00
Régis Behmo
7f7138f050 v13.1.11 (2022-04-12)
- [Security] Apply SAML security fix.
- [Improvement] In addition to the Docker build arguments
  `EDX_PLATFORM_REPOSITORY` and `NPM_REGISTRY`, also support two corresponding
  and identically-named `config.yml` values serving the same purpose.
2022-04-12 17:37:55 +02:00
Régis Behmo
16b2378165 sec: fix SAML vulnerability 2022-04-12 17:36:59 +02:00
Florian Haas
eaa578eabd feat: Make the platform repository and NPM registry configurable from config.yml
Previously, the only way for Tutor users to use a fork of edx-platform
or a custom NPM registry was to use build args during the image build.

This is suboptimal in the case of automatically building images from
CI pipelines, which may want to auto-detect when an image needs to be
rebuilt based on config.yml changes.

In addition, the EDX_PLATFORM_VERSION build argument can already be
set via a corresponding config.yml parameter (OPENEDX_COMMON_VERSION),
so it's reasonable to follow that precedent and also introduce
config.yml parameters to correspond with the EDX_PLATFORM_REPOSITORY
and NPM_REGISTRY build arguments.

Thus, introduce two new configuration parameters:

- EDX_PLATFORM_REPOSITORY
- NPM_REGISTRY

These parameters can now optionally be used instead of the
aforementioned build args.
2022-04-12 15:46:50 +02:00
Régis Behmo
11bfa96971 v13.1.10 (2022-04-11)
- [Feature] Upgrade all applications to open-release/maple.3.
2022-04-11 18:45:28 +02:00
Régis Behmo
0b86568ad5 feat: upgrade all apps to open-release/maple.3
Fresh out of the oven!
2022-04-11 18:44:15 +02:00
Rebecca Graber
7822f4e217 v13.1.9 (2022-04-06) 2022-04-06 22:56:40 +02:00
Emad Rad
a25ae73031 minor typos fixed 2022-03-29 08:44:33 +02:00
Régis Behmo
43c5177187 v13.1.8 (2022-03-18)
- [Bugfix] Fix "evalsymlink failure" during `k8s quickstart` (#611).
- [Bugfix] Fix "TypeError: upgrade() got an unexpected keyword argument 'non_interactive'" during `local upgrade`.
2022-03-24 09:41:41 +01:00
Régis Behmo
6e2cef0db5 fix: "evalsymlink failure" during k8s namespace creation
Close #611 (again).
2022-03-24 08:25:56 +01:00
Régis Behmo
8779bbd8b6 fix: TypeError during local quickstart
See: https://discuss.overhang.io/t/upgrade-error-upgrade-got-an-unexpected-keyword-argument-non-interactive/2582
2022-03-24 08:25:56 +01:00
Régis Behmo
3e2c4aebf4 v13.1.7 (2022-03-17)
- [Bugfix] Fix dockerize on arm64 by switching to the [powerman/dockerize](https://github.com/powerman/dockerize) fork (#591).
- [Bugfix] Fix "Unexpected args" error during service initialization on Kubernetes (#611).
2022-03-17 11:26:40 +01:00
Régis Behmo
a24b29271a fix: dockerize on arm64
The version of dockerize that shipped with the "openedx" image was not
compatible with arm64. The original project is unmaintained, but there
is a fork that provides a version that is compatible with arm64.

This was tested on arm64 with buildx:

    docker buildx build --tag=openedx --platform=linux/arm64 ~/.local/share/tutor/env/build/openedx

Close #591
2022-03-17 11:26:40 +01:00
Régis Behmo
f13c3db833 fix: "unexpected args" error on k8s
`kubectl_apply` was missing a "root" argument.

Close #611.
2022-03-17 10:40:57 +01:00
Régis Behmo
c63c952392 v13.1.6 (2022-03-15)
- [Bugfix] Fix `local/k8s quickstart` commands when upgrading from an older release (#595).
- [Bugfix] Fix running the default exim-relay SMTP server on arm64 (#600).
- [Feature] Add `tutor k8s apply` comand, which is a direct interface with `kubectl apply`.
- [Feature] Add `openedx-dockerfile-minimal` patch, which you can use to install custom packages and run commands as root in the Docker image.
2022-03-15 13:36:49 +01:00
Régis Behmo
653e4dc57d fix: "unexpected keyword argument" during quickstart upgrade
Close #595.
2022-03-15 13:35:44 +01:00
Emad Rad
dde22f3832 openedx-dockerfile-minimal added 2022-03-14 13:34:11 +01:00
Régis Behmo
49f11dd3a0 fix: upgrade exim-relay SMTP image to make it work on arm64
Close #600.
2022-03-14 12:45:17 +01:00
Régis Behmo
2d20a04363 feat: add a convenient tutor k8s apply command
This is convenient to check k8s template validity.
2022-02-25 14:50:36 +01:00
Régis Behmo
b78feec895 v13.1.5 (2022-02-14)
- [Improvement] Upgrade all services to open-release/maple.2.
2022-02-14 16:53:01 +01:00
Régis Behmo
156ef063ce feat: upgrade all services to open-release/maple.2 2022-02-14 14:38:43 +01:00
Régis Behmo
ec2598e114 v13.1.4 (2022-02-08)
- [Security] Fix vulnerability in redirect url during authentication (see
  [commit](06550411e3)).
2022-02-08 10:33:37 +01:00
Régis Behmo
c6cfa24dcc security: fix vulnerability in redirect url during auth 2022-02-08 09:52:28 +01:00
Régis Behmo
68b46aa720 v13.1.3 (2022-02-01)
- [Security] Fix vulnerability in call to invalid enrollment API (see [commit](e9369cffde)).
- [Bugfix] Fix "Internal Server Error / AttributeError / object has no attribute 'get_metadata'" in learning MFE.
- [Improvement] Replace all links to github.com/edx by github.com/openedx, following the migration of all repositories.
- [Bugfix] Fix `k8s start caddy` command.
2022-02-01 17:50:44 +01:00
Régis Behmo
89b4ce1202 security: fix enrollment API vulnerability 2022-02-01 17:40:53 +01:00
Régis Behmo
123786193e fix: object has no attribute 'get_metadata' error in learning MFE
See: https://github.com/openedx/edx-platform/pull/29741
2022-02-01 17:40:53 +01:00
Régis Behmo
9a4ea986e1 feat: migrate from edx to openedx GitHub org
edX has completed the migration of all repos from the "edx" to the "openedx"
organization. As a consequence, we change all the links in the repo.
2022-02-01 17:40:53 +01:00
Régis Behmo
bc87f78866 fix: k8s start caddy command
Caddy was not properly starting because its associated configmap was not
starting.

Close #577.
2022-01-31 08:37:35 +01:00
Régis Behmo
5958af86e3 v13.1.2 (2022-01-30)
- [Bugfix] Fix auto-renewal of certificates revoked by Let's Encrypt (see [discussion](https://community.letsencrypt.org/t/questions-about-renewing-before-tls-alpn-01-revocations/170449/21)).
2022-01-30 21:36:25 +01:00
Régis Behmo
7c3e54335d fix: let's encrypt auto-revoked certificates
Let's Encrypt started mass-revoking certificates. For some reason, we
did not upgrade to the latest version of Caddy in the Maple upgrade, so
we are affected by this issue. Users should upgrade now.

See:
https://discuss.overhang.io/t/lets-encrypt-revoked-certificate-did-anyone-experienced-this/2391
https://discuss.overhang.io/t/ssl-error-in-the-browser/2390
https://community.letsencrypt.org/t/questions-about-renewing-before-tls-alpn-01-revocations/170449/21
2022-01-30 21:33:31 +01:00
Régis Behmo
bbba7b6c4e v13.1.1 (2022-01-25)
- [Bugfix] Fix authentication in development due to missing SameSite policy on session ID cookie.
- [Bugfix] Display properly themed favicon.ico image in LMS, Studio and microfrontends.
- [Bugfix] Fix "LazyStaticAbsoluteUrl is not JSON serializable" error when sending bulk emails.
- [Bugfix] Fix `tutor local importdemocourse` fails when platform is not up.
2022-01-25 08:38:14 +01:00
Régis Behmo
26119983d4 fix: lms authenticaton in development
In development, it was no longer possible to authenticate to the lms. Ater
signing in, the session ID could not be dropped, and thus the user was not
signed in, although no error was logged -- just a warning in the browser
console.
This problem was caused by the fact that the SameSite policy was set to "None"
in development.
2022-01-25 08:36:52 +01:00
Régis Behmo
1172ab6f1f fix: display themed favicon across all apps
Previously, we were redirecting all /*favicon.ico requests to the default
favicon. This meant that the favicon might not necessarily be correctly themed,
most notably in MFEs. Here, we resolve this issue by redirecting to the
theme-agnostic theming/asset/* url. Also, we restrict the overly generic regexp
for favicon url matching. We verified that we did not miss any url by running
the following command on the demo server:

    tutor local logs caddy | grep --only-matching "host.*favicon.ico" | sort | uniq
2022-01-25 08:30:06 +01:00
Régis Behmo
405aaac189 fix: broken bulk emails because of LazyStaticAbsoluteUrl (again)
The LazyStaticAbsoluteUrl object was breaking bulk emails again with the
following stacktrace:

    2022-01-11 13:50:10,591 ERROR 12 [celery.app.trace] [user None] [ip None] trace.py:255 - Task lms.djangoapps.instructor_task.tasks.send_bulk_course_email[26b93357-018a-408f-b3f7-b69722447c5b] raised unexpected: EncodeError(TypeError('Object of type LazyStaticAbsoluteUrl is not JSON serializable'))
    Traceback (most recent call last):
      File "/openedx/venv/lib/python3.8/site-packages/kombu/serialization.py", line 50, in _reraise_errors
	yield
      File "/openedx/venv/lib/python3.8/site-packages/kombu/serialization.py", line 221, in dumps
	payload = encoder(data)
      File "/openedx/venv/lib/python3.8/site-packages/kombu/utils/json.py", line 69, in dumps
	return _dumps(s, cls=cls or _default_encoder,
      File "/openedx/venv/lib/python3.8/site-packages/simplejson/__init__.py", line 398, in dumps
	return cls(
      File "/openedx/venv/lib/python3.8/site-packages/simplejson/encoder.py", line 296, in encode
	chunks = self.iterencode(o, _one_shot=True)
      File "/openedx/venv/lib/python3.8/site-packages/simplejson/encoder.py", line 378, in iterencode
	return _iterencode(o, 0)
      File "/openedx/venv/lib/python3.8/site-packages/kombu/utils/json.py", line 59, in default
	return super(JSONEncoder, self).default(o)
      File "/openedx/venv/lib/python3.8/site-packages/simplejson/encoder.py", line 272, in default
	raise TypeError('Object of type %s is not JSON serializable' %
    TypeError: Object of type LazyStaticAbsoluteUrl is not JSON serializable

The point of that lazy object was to link to the lms logo even when a custom
theme was enabled. Luckily, we no longer need this lazy evaluation because we
now have theme-agnostic urls that point to static asset (see
https://github.com/openedx/edx-platform/pull/29461).

See:
https://discuss.overhang.io/t/error-while-sending-bulk-emails-lazystaticabsoluteurl-is-not-json-serializable/2176/
2022-01-25 08:27:36 +01:00
iamcristye@outlook.com
d953e99ae2 fix: tutor local importdemocourse fails when platform is not up 2022-01-20 11:51:24 +01:00
Régis Behmo
d45b36394a - [Improvement] Provide much more comprehensive instructions when
upgrading.
- [Bugfix] During upgrade, make sure that environment is up-to-date
prior to prompting to rebuild the custom images.
- [Bugfix] Fix ownership of mysql data, in particular when upgrading a
Kubernetes cluster to Maple.
- [Bugfix] Ensure that ``tutor k8s upgrade`` is run during ``tutor k8s
quickstart``, when necessary.
- 💥[Bugfix] By default, detect the current version during ``tutor
k8s/local upgrade``.
- [Bugfix] Fix upgrading from Lilac to Maple on Kubernetes by deleting
deployments and services.
2022-01-08 19:09:18 +01:00
Régis Behmo
4dc772d1e4 fix: attempt to make upgrade much clearer
`upgrade` had several issues, which are summarized here:
https://discuss.overhang.io/t/confusing-instructions-during-upgrade/2281/7

- The docs say that you should run quickstart, but what most people will see is
the big command tutor local upgrade --from=lilac verbatim paragraph.
- The local upgrade command should be very explicit about the fact that users
need to run quickstart.
- Maybe the name of the local upgrade command should be improved.
- When upgrading tutor from one major release to the next, there should be a
more explicit warning to inform users of what they are doing (see this other
conversation 1)
- We should tell people that they almost certainly need to enable the tutor and
the mfe plugins, if they are not enabled during upgrade.
- A link to all of the breaking changes from the changelog should be
prominently displayed during upgrade.
- The docs should emphasize that upgrading from one major release to the next
is potentially a risky endeavor and that downgrading is not possible. The docs
should also link to the changelog.

This commit has grown slightly beyond the intended scope, but the changes should be mostly positive.
2022-01-08 19:07:26 +01:00
Régis Behmo
9fc928a711 fix: update env prior to rebuilding images during upgrade
The fact that the environment was not up-to-date was causing errors and
confusion:
https://discuss.overhang.io/t/install-maple-importerror-cannot-import-name-removedindjango40warning/2255/6
2022-01-08 19:07:26 +01:00
Régis Behmo
59291eed52 fix: ownership of mysql data
In theory, we can assign ownership of mysql data to just any user. But in
Lilac, mysql was running with user 999. When upgrading to Maple, on Kubernetes,
the fsGroupChangePolicy was causing a change of the data *group* (to 1000) but
not of the user. This was causing a crash with the following error:

    [ERROR] InnoDB: The error means mysqld does not have the access rights to the directory.
2022-01-08 19:07:26 +01:00
Régis Behmo
70dbfcb6fb fix: upgrade from Lilac on k8s
When upgrading from Lilac, all services break with the following error:

    Service "***" is invalid: spec.ports[0].nodePort: Forbidden: may not be used when `type` is 'ClusterIP'

Upgrading deployments fails as well:

    Deployment.apps "***" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{"app.kubernetes.io/instance":"openedx-********", "app.kubernetes.io/managed-by":"tutor", "app.kubernetes.io/name":"***", "app.kubernetes.io/part-of":"openedx"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable

That's because deployments and services need to be deleted as part of the Maple
upgrade. So that's what we do as part of `tutor k8s upgrade --from=lilac`. And
we take the opportunity to:

1. Run upgrade as part of quickstart, when necessary.
2. Default to lilac during `tutor k8s upgrade`.

Close #551.
2022-01-08 19:07:26 +01:00
Régis Behmo
f4d866db81 v13.0.3 (2022-01-04)
- [Security] Upgrade Django to 3.2.11 in edx-platform.
- [Security] Prevent non-staff users from searching usernames by email by
  abusing the logout url.
2022-01-04 17:33:37 +01:00
Régis Behmo
699ef4c6d5 security: upgrade django to 3.2.11 in edx-platform
See release notes: https://docs.djangoproject.com/en/3.2/releases/3.2.11/
2022-01-04 16:38:23 +01:00
Régis Behmo
622e7cc097 security: prevent searching user emails by name (again)
This time, the security fix addresses an issue in the logout url.
2022-01-04 15:43:30 +01:00
Régis Behmo
94941d0387 v13.0.2
- [Security] Prevent non-staff users from searching usernames by email.
2021-12-22 00:46:13 +01:00
Régis Behmo
553000e5f5 security: fix search username by email
Without this patch, it is possible to search for account info including
username by using the email of a learner. This fix disallows searching using
email by regular users and restricts this feature to only staff and superusers.
2021-12-22 00:45:26 +01:00
Régis Behmo
4bfaa80bd3 v13.0.1
- [Fix] Missing requirements file in `pip install tutor[full]`.
2021-12-20 23:00:12 +01:00
Régis Behmo
14c40376ec fix: missing file in pip install tutor[full]
The missing requirement file is causing the install from pypi to fail.
2021-12-20 22:59:21 +01:00
Régis Behmo
7c157eccd5 feat: upgrade to Maple
- A shared cookie domain between lms and cms is no longer recommended:
https://github.com/edx/edx-platform/blob/master/docs/guides/studio_oauth.rst
- refactor: clean mounted data folder in lms/cms. In Lilac, the
bind-mounted lms/data and cms/data folders are a mess because new
folders are created there for every new course organisation.  These
folders are empty. As far as we know they are useless... With this
change we move these folders to a dedicated "modulestore" subdirectory;
which corresponds better to the initial intent of the fs_root setting.
- fix: frontend failure during login to the lms. See:
https://github.com/openedx/build-test-release-wg/issues/104
- feat: move all forum-related code to a dedicated plugin. Forum is an
optional feature, and as such it deserves its own plugin. Starting from
Maple, users will be able to install the forum from
https://github.com/overhangio/tutor-forum/
- migrate from DCS_* session cookie settings to SESSION_*. That's
because edx-platform no longer depends on django-cookies-samesite. Close
https://github.com/openedx/build-test-release-wg/issues/110
- get rid of tons of deprecation warnings in the lms/cms
- feat: make it possible to point to themed assets. Cherry-picking this
change makes it possible to point to themed assets with a theme-agnostic
url, notably from MFEs.
- Install all official plugins as part of the `tutor[full]` package.
- Don't print error messages about loading plugins during autocompletion.
- Prompt for image building when upgrading from one release to the next.
- Add `tutor local start --skip-build` option to skip building Docker images.

Close #450.
Close #545.
2021-12-20 21:21:36 +01:00
Braden MacDonald
1f46ed39c5 fix: openedx image wasn't building on ARM64 due to missing libgeos-dev 2021-12-14 12:10:49 +01:00
Régis Behmo
e89ff49223 v12.2.0
- 💥[Improvement] Fail on incorrect image name argument in `images
build/pull/push/printtag` commands.
- [Bugfix] Remove trailing slashes in docker-compose files for
[compatibility with docker-compose v2 in
WSL](https://github.com/docker/compose/issues/8558).
- [Improvement] `settheme` now works with preview domain.
- [Feature] Allow specifying extra pip packages through config.yml.
2021-12-08 11:29:52 +01:00
Régis Behmo
79c6d122e2 fix: incorrect "from" address in course bulk emails
Upstream PR: https://github.com/edx/edx-platform/pull/29001
See discussion:
https://discuss.overhang.io/t/sending-bulk-email-triggers-smtprecipientsrefused-error/1923
Related issue:
https://github.com/openedx/build-test-release-wg/issues/102
2021-12-08 11:29:52 +01:00
Régis Behmo
c3e6f57b08 feat: fail images ... [image] commands on incorrect image arg
Previously, `tutor images build doesnotexist` was a no-op. This was
counter-intuitive -- in particular, when users have a typo in their command.

Now, the `images build/pull/push/printtag` commands fail if the corresponding
image was not found.

See discussion: https://discuss.overhang.io/t/return-something-when-tutor-images-build-contains-a-typo/2174
2021-12-02 09:34:16 +01:00
Maxim Beder
19157d94bc feat: allow to specify extra pip packages in config
Added OPENEDX_EXTRA_PIP_REQUIREMENTS setting, which allows to specify
extra pip packages that should be installed.

Moved "openedx-scorm-xblock" package from Dockerfile to the new setting
in the config.yml.
2021-11-30 11:32:40 +01:00
iamcristye@outlook.com
5604cd4bf8 Improvement: settheme now works with preview domain 2021-11-29 18:21:16 +01:00
Sofiane Bebert
852759ae35 fix: remove trailing slashes in volume paths for docker-compose v2 compatibility
close #522
2021-11-22 14:10:28 +01:00
Régis Behmo
a4dcb9eaf2 v12.1.7 (2021-11-18)
- [Security] Timed exam security fix [29347](https://github.com/edx/edx-platform/pull/29347).
- [Feature] Add [tutor-richie](https://github.com/overhangio/tutor-richie) to the plugins that are bundled with the tutor binary.
- [Improvement] Make `tutor plugins list` print plugins sorted by name.
- [Improvement] Ignore Python plugins which cannot be loaded.
- [Bugfix] When configured with `RUN_FORUM: false`, omit forum-related [Jobs](https://kubernetes.io/docs/concepts/workloads/controllers/job/) from the manifests that `tutor k8s` generates. (#525)
2021-11-18 11:04:15 +01:00
Régis Behmo
aa931b33f4 security: applied timed exam fix 2021-11-18 10:55:50 +01:00
Florian Haas
0c3a160fce fix: Include the forum job only if RUN_FORUM is set
When running "tutor k8s" we can disable the forum deployment and
service by setting RUN_FORUM to false, but the corresponding job is
added to jobs.yml unconditionally.

Add a conditional to tutor/templates/k8s/jobs.yml so that the forum
job definition is only included when RUN_FORUM is true.

Fixes #525.
2021-11-18 10:22:54 +01:00
Régis Behmo
0a4a2bbac2 feat: add richie to bundled plugins
For more information, checkout the plugin docs:
https://github.com/overhangio/tutor-richie
2021-11-09 11:48:32 +01:00
Régis Behmo
485f47f6d0 fix: always print plugins sorted by name
`tutor plugins list` used to print plugins in random oredr. To be honest
this has always bothered me.
2021-11-08 10:40:37 +01:00
Régis Behmo
d9d08ad0f7 improvement: don't crash on unloadable plugins
In some cases, plugins declare an entry point but cannot be loaded.
This is the case when they depend on a version of tutor that is not the
one that is currently installed. This use case is very frequent when
working on multiple versions at the same time (i.e: right now, while we
are working on the Maple release). In such cases, it's best just to
ignore the plugin entirely rather than having to re-install all plugins
in the virtualenv.
2021-11-08 10:40:37 +01:00
Régis Behmo
0a2abe32dc v12.1.6 (2021-11-02)
- [Improvement] Upgrade all services to open-release/lilac.3.
- [Feature] Make it possible to override job configuration in
development: if they exist, `dev/docker-compose.jobs.yml` and
`dev/docker-compose.jobs.override.yml` will be loaded when running jobs.
- [Improvement] Faster `tutor local start` by building only necessary
images.
2021-11-02 15:55:10 +01:00
Régis Behmo
c9a08a5e18 chore: upgrade to open-release/lilac.3 2021-11-02 11:56:24 +01:00
Régis Behmo
d73d6732d5 feat: make it possible to override jobs in dev
Previously, job declarations were always loaded from local/docker-compose.yml
and local/docker-compose.jobs.yml. This meant that it was not possible to
override job declarations in dev mode. It is now the case, with
dev/docker-compose.jobs.yml and dev/docker-compose.jobs.override.yml. Neither
of these files exist yet... But who knows? we might need this feature one day.
In any case the code is much cleaner now.
2021-11-01 17:21:43 +01:00
Régis Behmo
079fb1c9ec fix: bypass build to accelerate "local start"
Previously, we were building all images every time we ran a "local start"
command. This was causing unnecessary rebuild. Here, instead, we make use of
the `docker-compose up --build`. This means that only the required images will
be rebuilt.
2021-11-01 17:00:11 +01:00
Peter Parkkali
2549aef4dc fix: require at least 4 GB RAM on macOS for 'local quickstart' only
Limits the memory chek to the 'local quickstart' command, makes error
handling more accurate and adds warning messages for some conditions.
Also adds a mention of this in troubleshooting.rst.
2021-10-25 20:17:38 +02:00
Peter Parkkali
fb2aeefd91 fix: require at least 4 GB RAM on macOS for local commands
Adds a check in the 'local' command group that requires at least
4 GB of RAM to be allocated to Docker when running any of the
local subcommands on macOS. This addresses a common issue where
Docker's default setting (2 GB) causes startup to crash with
misleading error messages.
2021-10-25 20:17:38 +02:00