6
0
mirror of https://github.com/ChristianLight/tutor.git synced 2024-11-14 17:24:08 +00:00
Commit Graph

592 Commits

Author SHA1 Message Date
Régis Behmo
e7d3e722ce fix: "upstream sent too big header" nginx errors
It was reported that existing users could not sign-in after a Koa -> Lilac
upgrade. This is due to the fact that cookies exceed the 4kb limit, which is
rejected by nginx. The fix consists in increasing this limit to 8kb:
http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_buffer_size

This issue was reported here: https://discuss.overhang.io/t/lilac-upgrade-login-issues/1617
Note that I was not able to reproduce the issue. The fix was tested by the
issue reporter.
2021-06-22 08:58:50 +02:00
Régis Behmo
ceddc11c29 feat: upgrade to open-release/lilac.master
One of the breaking changes of this release is the removal of the webui and
android features; these are moved to dedicated plugins. This causes a breaking
change: the renaming of the DOCKER_IMAGE_ANDROID
config variable to ANDROID_DOCKER_IMAGE.

See this TEP for reference: https://discuss.overhang.io/t/separate-webui-and-android-from-tutor-core-and-move-to-dedicated-plugins/1473
2021-06-08 23:29:12 +02:00
Régis Behmo
915551268c v11.3.1 (2021-06-08)
- [Improvement] Avoid permission issues in Kubernetes/Openshift for users who do not have the rights to edit their namespace.
- [Improvement] Better Kubernetes object creation.
2021-06-08 20:17:44 +02:00
Régis Behmo
4a451844e6 fix: avoid namespace edition for k8s users without access rights
In most cases, it makes very little sense to edit the namespace that an
application is running in. Quite often, users are granted access to just one
namespace and don't have the necessary rights to edit the namespace -- and for
good security reasons. In such cases, the k8s namespace object already exists
and there is no need for the user to edit or create it. Here, what we do is
that we create the namespace only if it does not exist. This should solve quite
a few permission issues, notably for Openshift users.
2021-06-08 19:54:09 +02:00
Régis Behmo
9b6b770dd7 feat: better object creation in "k8s start"
When running "tutor k8s start", it's unnecessary to attempt to create volumes
and namespace multiple times.
2021-06-08 19:54:09 +02:00
Régis Behmo
900572fdae v11.3.0 (2021-05-18)
- 💥[Security] Disable python-evaluated input by default as we don't run codejail.
- [Bugfix] Fix missing discussion notifications.
- [Improvement] Better error logging when loading a plugin from an incompatible version.
2021-06-06 14:40:24 +02:00
Régis Behmo
a79d194bbc security: disable python-evaluated input by default
As described in issue #284, tutor does not come with codejail enabled out of
the box. Actually, we don't even have a working plugin, yet. To prevent users
from running unsafe code, we explicitely disable python-evaluated input by
disabling the "python" interpreter. This might break some courses; thus, this
is a non-backward compatible change.
2021-06-06 14:37:46 +02:00
liupengcheng
58c77ad3fb fix: discussion email notification not work.
Discussion email notification relies on edx_ace and celery routing
key for ace is not set properly by default, which causes the send
mail task is never consumed.

Close #439.
2021-05-25 10:00:45 +02:00
Régis Behmo
4cc48c966f feat: better logging during plugin loading failure
When upgrading Tutor plugins to the next release, I often end up with a
virtualenv that contains plugins that depend on different versions of
tutor-openedx. This causes a crash that did not log the name of the responsible
package. For instance:

    Traceback (most recent call last):
      File "/home/regis/venvs/tutor/bin/tutor", line 11, in <module>
        load_entry_point('tutor-openedx', 'console_scripts', 'tutor')()
      File "/home/regis/projets/overhang/repos/overhang/tutor/tutor/commands/cli.py", line 37, in main
        add_plugin_commands(cli)
      File "/home/regis/projets/overhang/repos/overhang/tutor/tutor/commands/plugins.py", line 137, in add_plugin_commands
        for plugin in plugins.iter_installed():
      File "/home/regis/projets/overhang/repos/overhang/tutor/tutor/plugins.py", line 401, in iter_installed
        yield from Plugins.iter_installed()
      File "/home/regis/projets/overhang/repos/overhang/tutor/tutor/plugins.py", line 362, in iter_installed
        for plugin in PluginClass.iter_installed():
      File "/home/regis/projets/overhang/repos/overhang/tutor/tutor/plugins.py", line 199, in iter_installed
        for plugin in cls.iter_load():
      File "/home/regis/projets/overhang/repos/overhang/tutor/tutor/plugins.py", line 233, in iter_load
        yield cls(entrypoint)
      File "/home/regis/projets/overhang/repos/overhang/tutor/tutor/plugins.py", line 220, in __init__
        super().__init__(entrypoint.name, entrypoint.load())
      File "/home/regis/venvs/tutor/lib/python3.8/site-packages/pkg_resources/__init__.py", line 2442, in load
        self.require(*args, **kwargs)
      File "/home/regis/venvs/tutor/lib/python3.8/site-packages/pkg_resources/__init__.py", line 2465, in require
        items = working_set.resolve(reqs, env, installer, extras=self.extras)
      File "/home/regis/venvs/tutor/lib/python3.8/site-packages/pkg_resources/__init__.py", line 791, in resolve
        raise VersionConflict(dist, req).with_context(dependent_req)
    pkg_resources.VersionConflict: (tutor-openedx 11.2.11 (/home/regis/projets/overhang/repos/overhang/tutor), Requirement.parse('tutor-openedx<13.0.0,>=12.0.0'))

In this commit, we introduce an error log that displays the name and location
of the package. E.g:

    Failed to load entrypoint 'minio = tutorminio.plugin' from distribution tutor-minio 12.0.0
2021-05-20 12:22:52 +02:00
Régis Behmo
ed1e2e49fe v11.2.11 (2021-05-18)
- [Feature] Add redis database configuration for both cache and celery.
2021-05-18 10:05:21 +02:00
liupengcheng
028162fea4 feat: Make openedx redis db configurable.
Redis db 0 is used for celery broker and db 1 is used for cache
by default.
2021-05-18 09:35:36 +02:00
Régis Behmo
5658b46715 v11.2.10 (2021-05-17)
- [Security] Apply Django security patches by upgrading from 2.2.20 to 2.2.23.
- [Bugfix] Fix video unit completion (see [pull
  request](https://github.com/edx/edx-platform/pull/27230)).
2021-05-17 21:29:52 +02:00
Régis Behmo
26d14457ec security: upgrade Django to 2.2.23
Also, fix video unit completion in LMS.
2021-05-17 14:25:13 +02:00
Régis Behmo
306d270a3a v11.2.9 (2021-05-12)
- [Bugfix] Fix crashing installation because of a major release of all
Pallets projects.
- [Bugfix] Fix crash in `local quickstart -p` command.
- [Bugfix] Fix 502 error on request to lms with header larger than the
maximum uwsgi buffer size.
2021-05-12 10:14:27 +02:00
Régis Behmo
1a5e2e005b fix: requirements of pallets project
All pallets project requirement had a major upgrade today:
https://palletsprojects.com/blog/flask-2-0-released/

We are not yet compatible with click 8.0 and others. In particular,
click-repl imports modules which are no longer available. Until we can
upgrade, we add constraints to the requirements files.

The following error was being raised:

    $ tutor plugins list
    Traceback (most recent call last):
      File "/home/data/regis/tmp/testtutor/bin/tutor", line 5, in <module>
	from tutor.commands.cli import main
      File "/home/data/regis/tmp/testtutor/lib/python3.6/site-packages/tutor/commands/cli.py", line 6, in <module>
	import click_repl
      File "/home/data/regis/tmp/testtutor/lib/python3.6/site-packages/click_repl/__init__.py", line 6, in <module>
	import click._bashcomplete
    ModuleNotFoundError: No module named 'click._bashcomplete'

Close #430.
2021-05-12 10:12:31 +02:00
Régis Behmo
6d92fe2d4a fix: crash during local quickstart -p
When running `tutor local quickstart -p` we were getting the following error:

    Usage: custom [OPTIONS] ARGS...
    Try 'custom --help' for help.

    Error: Missing argument 'ARGS...'.

The docker-compose command sometimes accept a single command ("pull") with zero
argument.

See: https://discuss.overhang.io/t/local-quickstart-not-working-when-pullimages-enabled/1526
2021-05-07 17:01:09 +02:00
wey-gu
452cd229f3 fix: 502 error on request to lms with large header
According to [1], request to uwsgi with header
larger than the default value 4096 will end up
with below error as #426:
  invalid request block size: 4123 (max 4096)...skip
  hr_instance_read(): Connection reset by peer
  [plugins/http/http.c line 647]

This commit fixes it by changing the maximum
buffer size as 4096

Also: Similar issue was already identifed and
fixed in [2] on tutor-ecommerce

[1] https://uwsgi-docs.readthedocs.io/en/latest/ThingsToKnow.html
[2] 6df2c99362

Close #426
2021-05-01 16:32:19 +02:00
Régis Behmo
dfd3d179e9 v11.2.8 (2021-04-27)
- [Bugfix] Fix parsing of YAML-formatted values in ``config save --set
  KEY=VALUE`` commands, in use for instance with Ecommerce.
2021-04-27 09:26:01 +02:00
Eric Herrera
389dd96fdd fix: YamlParamType supports line terminators
This fix allows using a multiple line formatted Yaml string as input for setting a Tutor config value.
2021-04-25 09:14:21 +02:00
Régis Behmo
2e0f136a23 v11.2.7 (2021-04-23)
- [Security] Apply security patch
  [27394](https://github.com/edx/edx-platform/pull/27394).
- [Feature] Add patches to extend python requirements installation process in
  openedx and openedx-dev Dockerfiles.
- [Improvement] Apply edx-platform patches during Docker image build using
  tutor patch 'openedx-dockerfile-git-patches-default'.
2021-04-23 20:24:08 +02:00
Régis Behmo
b211cb0a0f security: apply security patch 27394
See: https://discuss.openedx.org/t/security-patch-for-xss-fixes/4820
2021-04-23 20:23:11 +02:00
Eric Herrera
17088e2fc6 feat: Conditional edx-platform patching
During Docker images build process, apply custom edx-platform patches when tutor patch 'openedx-dockerfile-git-patches-default' is defined or apply current release patches in other case. It avoids possible conflicts between the actually used edx-platform version and the current release patches.
2021-04-15 07:32:39 +02:00
Eric Herrera
a05486e7b0 feat: openedx Dockerfile python requirements extension patch
Add patches to extend python requirements installation process in openedx and openedx-dev Dockerfiles
2021-04-15 07:32:39 +02:00
Régis Behmo
e532f34909 v11.2.6 (2021-04-09)
- [Improvement] Upgrade all services to open-release/koa.3.
- [Feature] Make it possible to build the openedx Docker image with a custom openedx-i18n version with the ``--build-arg OPENEDX_I18N_VERSION=custom`` command line argument.
2021-04-09 10:16:17 +02:00
Régis Behmo
4a65f24ae7 feat: upgrade all services to open-release/koa.3
We remove security patches and custom fixes which are now part of koa.3.

We take the opportunity to make it possible to build the openedx Docker image
without relying on a corresponding openedx-i18n repo tag: often, we want to
test whether the image simply builds successfully, and we don't need up-to-date
translations. For those cases, it's now possible to pass the `-a
OPENEDX_I18N_VERSION=oldertag` build argument.
2021-04-09 10:13:34 +02:00
Régis Behmo
f2ab27199e v11.2.5 (2021-03-30)
- [Bugfix] Fix edge case where `PLUGINS` entry is null in config.yml.
- [Bugfix] Fix missing py2neo dependency in `images build openedx` (#411).
2021-03-30 09:25:39 +02:00
Régis Behmo
c01f4476b8 fix: TypeError when PLUGINS is None
When the PLUGINS config entry is None (`PLUGINS:`), the following error
was being triggered:

  File "/.../tutor/tutor/plugins.py",
  line 304, in is_enabled
      return name in config.get(CONFIG_KEY, [])
      TypeError: argument of type 'NoneType' is not iterable
2021-03-30 09:23:16 +02:00
Régis Behmo
8db1495497 fix: missing py2neo dependency breaks images build openedx
All releases of py2neo prior to 4.0.0 were stripped from Pypi, so we
need to install this dependency from source.

See:
https://github.com/openedx/build-test-release-wg/issues/37
https://github.com/edx/edx-platform/pull/27168

Close #411.
2021-03-30 09:13:44 +02:00
Régis Behmo
98ed40c800 v11.2.4 (2021-03-17)
- [Bugfix] Fix "Invalid Request" error during SAML authentication
(thanks @Ant1x!).
- [Feature] Add `images build --pull` option to update base images.
- [Improvement] Annotate types all over the Tutor code base.
- [Bugfix] Fix parsing of YAML CLI arguments that include equal "="
signs.
- [Bugfix] Fix minor edge case in `long_to_base64` utility function.
- [Improvement] Add openedx patches to add settings during build
process.
2021-03-17 18:21:16 +01:00
Régis Behmo
f688a279dc Actually remove the images build --pull option
This option is mostly useless to us in CI, as it attempts (and fails) to
update the base image when building the dev image. For good reason: the
base image of the dev image is the latest openedx, which has not been
pushed to docker.io at the time of building -_-
2021-03-17 18:21:16 +01:00
Régis Behmo
b00bae9e72 minor: add missing changelog entry 2021-03-17 17:33:06 +01:00
Régis Behmo
914dbc53be Add images build --pull option to pull base images on build
This will be useful in CI, where we want to automatically update base
images as soon as a security fix comes out.
2021-03-17 16:02:46 +01:00
Régis Behmo
0a670d7ead refactor: add type annotations
Annotations were generated with pyannotate:
https://github.com/dropbox/pyannotate

We are running in strict mode, which is awesome!

This affects a large part of the code base, which might be an issue for
people running a fork of Tutor. Nonetheless, the behavior should not be
affected. If anything, this process has helped find and resolve a few
type-related bugs. Thus, this is not considered as a breaking change.
2021-03-15 21:46:55 +01:00
Eric Herrera
6ca863e04c Add patches that enable adding settings during build process (collectsatic and i18ncompile). 2021-03-08 13:09:53 +01:00
Régis Behmo
070b75caa4 fix: parsing of YAML config params with "=" signs
Close #405.
2021-03-08 12:24:40 +01:00
Régis Behmo
de872282b2 fix: minor edge case in long_to_base64 utility function.
This was detected by mypy.
2021-02-25 09:54:46 +01:00
Régis Behmo
59e368a449 v11.2.3 (2021-02-20)
- [Bugfix] Make LMS celery workers actually process LMS tasks, and not
  CMS tasks.
2021-02-20 12:50:49 +01:00
Régis Behmo
89dae24b83 minor changelog formatting
Sentences end with a dot ".".
2021-02-20 12:49:30 +01:00
Régis Behmo
39924121be fix: make LMS process LMS tasks, and not CMS tasks
lms-worker was configured to run CMS tasks instead of LMS tasks. I'm not
sure what tasks were being dismissed, and what is the actual production
impact.
2021-02-20 12:45:17 +01:00
Régis Behmo
6c72877256 v11.2.2 (2021-02-17)
- [Security] Apply security patch [26592](https://github.com/edx/edx-platform/pull/26592)
2021-02-17 20:01:53 +01:00
Régis Behmo
8ad45e6157 Apply security patch 26592
See: https://discuss.openedx.org/t/security-im-patch-for-xss-fixes-9/4297
2021-02-17 20:00:44 +01:00
Régis Behmo
8932ea0541 v11.2.1 (2021-02-10)
- [Bugfix] Actually persist Redis data
2021-02-16 12:40:04 +01:00
Régis Behmo
28afa890bf fix: actually persist redis data
Redis data was not actually persisted, because the redis configuration file was
not mounted from the right location. In order to mount redis data in a
host-mounted directory, the working directory has to be properly set.

The problem was occurring both with docker-compose and Kubernetes.

Close #404.
2021-02-16 12:26:54 +01:00
Régis Behmo
731946cce6 v11.2.0 (2021-02-10)
- [Bugfix] Upgrade all services to open-release/koa.2
2021-02-10 00:18:00 +01:00
Régis Behmo
02f9d8db44 Upgrade to open-release/koa.2 2021-02-10 00:15:54 +01:00
Régis Behmo
1d5886baee v11.1.5 (2021-02-009)
- [Security] Apply security patch [26432](https://github.com/edx/edx-platform/pull/26432).
- [Bugfix] Print warnings to stderr
2021-02-09 09:28:39 +01:00
Régis Behmo
9c559cd30b Apply security patch 26432
See: https://github.com/edx/edx-platform/pull/26432/commits
2021-02-09 09:26:43 +01:00
Régis Behmo
73dfb5a616 Print warnings to stderr
When tutor is run as root, the `tutor config printroot` and `tutor
config printvalue` commands used to write to stdout, which was causing
issues when combined with `tutor config render` and `tutor local
settheme` commands.

See discussion:
https://discuss.overhang.io/t/error-while-install-indigo-theme/1056
2021-02-05 15:44:19 +01:00
Régis Behmo
6bed4f6d4a v11.1.4 (2021-02-04)
- [Security] Apply security patch
  [26358](https://github.com/edx/edx-platform/pull/26358).
2021-02-04 17:02:40 +01:00
Régis Behmo
dca4eb6a32 Apply security patch 26358 2021-02-04 08:55:26 +01:00
Régis Behmo
0f5feeb4eb v11.1.3 (2021-01-31)
- [Security] Apply security patch
  [26253](https://github.com/edx/edx-platform/pull/26253).
2021-01-31 10:58:46 +01:00
Régis Behmo
ed66ccdcf4 Apply security patch #26029
See: https://discuss.openedx.org/t/security-im-patch-for-xss-fixes-6/4162
2021-01-31 10:57:53 +01:00
Régis Behmo
56be99f854 v11.1.2 (2021-01-29)
- [Bugfix] Fix django setting value and static asset loading in openedx-dev
  image
2021-01-29 08:44:22 +01:00
Adolfo R. Brandes
d4b067483d Fix default settings on development environment
First, allow using custom Django settings on a development
environment (as documented but not implemented), setting it to the
correct value of `tutor.development`.  Prior to this, `tutor dev
runserver lms` would default to `tutor.production` when on a custom edX
branch.

Second, fix the documentation so the correct environment variable is
described, at the same time removing an option that doesn't seem to work.

See discussion: https://discuss.overhang.io/t/koa-dev-lms-doesnt-find-static-content/1250
2021-01-29 08:43:46 +01:00
Régis Behmo
6fa1db1e4f v11.1.1 (2021-01-20)
- [Feature] Add a `tutor images build --target=...` argument for [multi-stage
  Docker
builds](https://docs.docker.com/develop/develop-images/multistage-build/).
- [Feature] Create a test version of the openedx-dev Docker image for running
  edx-platform unit tests.
- [Security] Apply security patch
  [26112](https://github.com/edx/edx-platform/pull/26112)
- [Bugfix] Fix `local exec` command which crashed with a `AttributeError`.
2021-01-20 14:38:08 +01:00
Régis Behmo
8d803fb08b Get edx-platform unit tests to run
We manage to get unit tests to run in a dedicated openedx-test container. Only
35 tests are failing (out of 17k). I suspect these tests are also failing in
the devstack.
2021-01-20 14:37:10 +01:00
Régis Behmo
3ab800f2e2 Apply security patch 26112 2021-01-20 14:35:47 +01:00
Régis Behmo
df3d432518 Fix AttributeError on "local exec"
`local exec` was failing with the following stacktrace:

    Traceback (most recent call last):
      File "/home/regis/venvs/tutor/bin/tutor", line 11, in <module>
        load_entry_point('tutor-openedx', 'console_scripts', 'tutor')()
      File "/home/regis/projets/overhang/repos/overhang/tutor/tutor/commands/cli.py", line 38, in main
        cli()  # pylint: disable=no-value-for-parameter
      File "/home/regis/venvs/tutor/lib/python3.8/site-packages/click/core.py", line 829, in __call__
        return self.main(*args, **kwargs)
      File "/home/regis/venvs/tutor/lib/python3.8/site-packages/click/core.py", line 782, in main
        rv = self.invoke(ctx)
      File "/home/regis/venvs/tutor/lib/python3.8/site-packages/click/core.py", line 1259, in invoke         return _process_result(sub_ctx.command.invoke(sub_ctx))
      File "/home/regis/venvs/tutor/lib/python3.8/site-packages/click/core.py", line 1259, in invoke         return _process_result(sub_ctx.command.invoke(sub_ctx))
      File "/home/regis/venvs/tutor/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
        return ctx.invoke(self.callback, **ctx.params)                                                     File "/home/regis/venvs/tutor/lib/python3.8/site-packages/click/core.py", line 610, in invoke
        return callback(*args, **kwargs)                                                                   File "/home/regis/projets/overhang/repos/overhang/tutor/tutor/commands/compose.py", line 237, in ex
    ecute                                                                                                    dc_command.callback("exec", args)
      File "/home/regis/venvs/tutor/lib/python3.8/site-packages/click/decorators.py", line 33, in new_func
        return f(get_current_context().obj, *args, **kwargs)
      File "/home/regis/projets/overhang/repos/overhang/tutor/tutor/commands/compose.py", line 271, in dc_command
        volumes, non_volume_args = bindmounts.parse_volumes(args)                                          File "/home/regis/projets/overhang/repos/overhang/tutor/tutor/bindmounts.py", line 73, in parse_vol
    umes
        context = custom_docker_compose.make_context("custom", args)
      File "/home/regis/venvs/tutor/lib/python3.8/site-packages/click/core.py", line 700, in make_context    self.parse_args(ctx, args)
      File "/home/regis/venvs/tutor/lib/python3.8/site-packages/click/core.py", line 1045, in parse_args
        opts, args, param_order = parser.parse_args(args=args)
      File "/home/regis/venvs/tutor/lib/python3.8/site-packages/click/parser.py", line 269, in parse_args
        self._process_args_for_options(state)
      File "/home/regis/venvs/tutor/lib/python3.8/site-packages/click/parser.py", line 289, in _process_a
    rgs_for_options
        arg = state.rargs.pop(0)
    AttributeError: 'tuple' object has no attribute 'pop'
2021-01-19 11:51:37 +01:00
Régis Behmo
b648e413e6 v11.1.0 (2021-01-13)
- [Bugfix] Fix "Read-only file system" errors when running `tutor dev
  runserver` commands on Mac OS. (again, see #392)
- [Feature] Automatically bind-mount volumes from the `volumes/` directory with
  the `--volume=/...` syntax.
2021-01-13 12:32:48 +01:00
Régis Behmo
b09d2ea1ed Fix "Read-only file system" error in dev (again)
Looks like the `-perm /u=w` argument incorrectly includes files from the
read-only filesystem. This should be fixed by the `-writable` option.

Close #392
2021-01-13 12:27:34 +01:00
Régis Behmo
1c927c6e96 Automatically bind-mount volumes from volumes/
This introduces a new dev/local command:

    tutor dev bindmount CONTAINER PATH

And a new volume syntax:

    tutor dev run --volume=PATH CONTAINER

This syntax automatically bind-mounts folders from the tutorroot/volumes
directory, which is pretty nifty.
2021-01-12 22:43:06 +01:00
Régis Behmo
b015890857 v11.0.7 (2021-01-11)
- [Security] Apply security patch [26029](https://github.com/edx/edx-platform/pull/26029)
2021-01-11 14:20:42 +01:00
Régis Behmo
0435483e14 Apply security patch #26029
See: https://discuss.openedx.org/t/security-im-patch-for-xss-fixes-4/3997
2021-01-11 14:19:50 +01:00
Régis Behmo
d2ec64fe70 v11.0.6 (2021-01-05)
- [Security] Apply security patch
[25974](https://github.com/edx/edx-platform/pull/25974)
2021-01-05 14:06:07 +01:00
Régis Behmo
813a83b029 Apply security patch #25974
See https://discuss.openedx.org/t/security-im-patch-for-xss-fixes-3/3944
2021-01-05 14:05:31 +01:00
Régis Behmo
75af640dbf v11.0.5 (2020-12-30)
- [Bugfix] Fix "Invalid type for parameter ContentType" error on js
upload in Scorm xblock ([openedx-scorm-xblock #16](https://github.com/overhangio/openedx-scorm-xblock/issues/16))
2020-12-30 16:17:34 +01:00
Régis Behmo
3bdc038971 Fix "Invalid type for parameter ContentType" error in scorm xblock
Close https://github.com/overhangio/openedx-scorm-xblock/issues/16
See upstream PR: https://github.com/edx/edx-platform/pull/25957
See discussion: https://discuss.overhang.io/t/issue-with-scorm/1126
2020-12-30 12:21:55 +01:00
Régis Behmo
1c9e112114 v11.0.4 (2020-12-17)
- [Bugfix] Fix "Read-only file system" errors when running `tutor dev
runserver` commands on Mac OS. (#392)
2020-12-17 19:12:30 +01:00
Régis Behmo
661d238600 Fix "Read-only file system" errors
Close #392.
2020-12-17 19:10:42 +01:00
Régis Behmo
09ee675ae2 v11.0.3 (2020-12-15)
- [Bugfix] Fix upload of video transcripts to S3 (again)
2020-12-15 12:54:01 +01:00
Régis Behmo
d45c354191 Fix upload of video transcripts to S3 (again) 2020-12-15 12:53:23 +01:00
Régis Behmo
189cefa5f7 v11.0.2 (2020-12-12)
- [Bugfix] Fix missing celery tasks from edx-platform (see [upstream
PR](https://github.com/edx/edx-platform/pull/25840))
2020-12-12 11:49:09 +01:00
Régis Behmo
4d9b85a849 Fix missing celery tasks in edx-platform
See: https://github.com/edx/edx-platform/pull/25840
2020-12-12 11:48:39 +01:00
Régis Behmo
88814c5457 v11.0.1 (2020-12-10)
- [Security] Apply security patch
[25834](https://github.com/edx/edx-platform/pull/25834)
- [Bugfix] Fix Android apk directory mount path
2020-12-10 13:21:58 +01:00
Régis Behmo
1394090a48 Apply security patch
See:
https://github.com/edx/edx-platform/pull/25834/
https://discuss.openedx.org/t/security-im-patch-for-xss-fixes-2/3802
2020-12-10 13:21:25 +01:00
Régis Behmo
08e2a30e6b Fix Android apk directory mount path 2020-12-10 11:16:32 +01:00
Régis Behmo
728ef966dc v11.0.0 (2020-12-09)
- 💥[Improvement] Upgrade Open edX to Koa
- 💥 Setting changes:
    - The ``ACTIVATE_HTTPS`` setting was renamed to ``ENABLE_HTTPS``.
    - Other ``ACTIVATE_*`` variables were all renamed to ``RUN_*``.
    - The ``WEB_PROXY`` setting was removed and ``RUN_CADDY`` was added.
    - The ``NGINX_HTTPS_PORT`` setting is deprecated.
- Architectural changes:
    - Use Caddy as a web proxy for automated SSL/TLS certificate generation:
	- Nginx no longer listens to port 443 for https traffic
	- The Caddy configuration file comes with a new ``caddyfile`` patch for much simpler SSL/TLS management.
	- Configuration files for web proxies are no longer provided.
	- Kubernetes deployment no longer requires setting up a custom Ingress resource or custom manager.
    - Gunicorn and Whitenoise are replaced by uwsgi: this increases boostrap performance and makes it no longer necessary to mount media folders in the Nginx container.
    - Replace memcached and rabbitmq by redis.
- Additional features:
    - Make it possible to disable all plugins at once with ``plugins disable all``.
    - Add ``tutor k8s wait`` command to wait for a pod to become ready
    - Faster, more reliable static assets with local memory caching
- Deprecation: proxy files for Apache and Nginx are no longer provided out of the box.
- Removed plugin `{{ patch (...) }}` statements:
    - "https-create", "k8s-ingress-rules", "k8s-ingress-tls-hosts": these are no longer necessary. Instead, declare your app in the "caddyfile" patch.
    - "local-docker-compose-nginx-volumes": this patch was primarily used to serve media assets. The recommended is now to serve assets with uwsgi.
2020-12-10 01:05:02 +01:00
Régis Behmo
d3c842c8da v10.5.3 (2020-12-09)
- [Security] Apply upstream edx-platform [security
patch](https://github.com/edx/edx-platform/pull/25782)
2020-12-09 21:50:42 +01:00
Régis Behmo
0f44d32147 Apply upstream security patch 2020-12-09 13:34:14 +01:00
Régis Behmo
0fc1358eee v10.5.2 (2020-12-07)
- [Improvement] Increase the timeout of the gunicorn worker command in
openedx Dockerfile
2020-12-07 13:23:35 +01:00
Maarten de Waard
b42beb83d9 Increase the timeout of the gunicorn worker command in openedx Dockerfile
As discussed in
https://discuss.overhang.io/t/lms-worker-timeout-issue/1057/8, the
timeout of the gunicorn worker is a bit low, causing the worker to
crashloop on slower systems. By increasing the timeout to 120 seconds,
the worker should have time to come up before it gets restarted.
2020-12-07 13:21:41 +01:00
Régis Behmo
790aa2b2d2 v10.5.1 (2020-11-30)
- [Bugfix] Fix Dockerfile parsing on Windows
- [Improvement] Add option to patch lms and cms nginx server blocks
2020-11-30 15:42:19 +01:00
Maarten de Waard
5aa78273cf add option to patch lms and cms nginx server block 2020-11-30 15:34:43 +01:00
Régis Behmo
727c204e92 Fix Dockerfile parsing on Windows
Line endings are written as "\r" by default on Windows, which makes the
Dockerfiles unreadable by Docker.

See: https://github.com/overhangio/tutor/pull/385#issuecomment-730387969
2020-11-20 16:05:56 +01:00
Régis Behmo
7b16af22b8 v10.5.0 (2020-11-19)
- 💥[Improvement] Remove `dev/local pullimages`. Instead, run `dev/local
dc pull`.
- 💥[Improvement] Add `dev dc` and `local dc` commands as thin wrappers
of the `docker-compose` CLI.
- 💥[Improvement] Remove the undocumented `local run_hook` command.
Instead, run `local init --limit=...`.
- 💥[Improvement] Remove `tutor android pullimage` command. Instead, run
`tutor images pull android`.
- [Bugfix] Fix `config save` on Windows.
- [Bugfix] Fix platform checking in user ID function
2020-11-19 11:32:17 +01:00
Régis Behmo
cacb4f4cfd Remove dev/local pullimages command
Now that we have the `dc` command this can be simplified to `dc pull`.
2020-11-16 12:50:01 +01:00
Régis Behmo
3332f9a330 Add dev dc and local dc commands
These are thin wrappers of `docker-compose` -- with all the right
arguments in place.
2020-11-16 12:46:01 +01:00
Régis Behmo
9a9ddf5497 Remove the undocumented run_hook command
This undocumented command was superseded by `local init --limit=...`.
2020-11-16 12:22:28 +01:00
Régis Behmo
df2e977b35 Remove android pullimage command
Instead, run `images pull android`.
2020-11-16 12:19:53 +01:00
Régis Behmo
b2fdaf604e Fix platform checking and env saving on Windows 2020-11-12 12:56:18 +01:00
Régis Behmo
2596e5f7d2 v10.4.1 (2020-11-11)
- [Bugfix] Fix dependency error during `pip install tutor` due to urllib3 incompatibility
- [Bugfix] Fix user ID checking under Windows.
- [Bugfix] Fix template rendering for Windows users.
- [Improvement] Switch to `bcrypt` for htpasswd password generation, for better portability on Windows.
- [Improvement] In the openedx production docker image, add some jitter to the gunicorn worker restart process to prevent all workers from restarting at the same time.
2020-11-11 10:39:58 +01:00
Régis Behmo
ad858cd699 Fix urllib3 dependency error on installation of tutor
requests depends on urllib3<1.26.0, while urllib3==1.26.0 was just
released. We need to introduce a constraint on urllib3 to avoid the
following error when running `pip install -e`:

    pkg_resources.ContextualVersionConflict: (urllib3 1.26.0
    (./venv/lib/python3.7/site-packages),
    Requirement.parse('urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1'),
    {'requests'})

 We will be able to remove this constraint once requests becomes
 compatible with urllib3==1.26.0. To test this change just run `pip
 install -e tutor` and then run a dummy command, such as `tutor local
 stop`.
2020-11-11 10:36:42 +01:00
Régis Behmo
6c4d9619fd Fix user ID checking under windows
This should make tutor portable on Windows. Well, if the `id -u` command
is supported at least...

See: https://github.com/overhangio/tutor/issues/381
2020-11-07 17:48:20 +01:00
Régis Behmo
998dec7149 Fix template name separator for Windows users
See: https://github.com/overhangio/tutor/issues/381
2020-11-07 16:37:43 +01:00
Régis Behmo
6848253b8e Switch from crypt to bcrypt for htpasswd generation
This is more portable, as described here:
https://github.com/overhangio/tutor/issues/381
2020-11-07 14:46:53 +01:00
Régis Behmo
02cc106af3 Add some jitter to the openedx gunicorn workers
This should prevent gunicorn workers from restarting all at the same
time.
2020-11-04 18:19:35 +01:00
Régis Behmo
91e647692a Add upgrade instructions for AMI users
AWS unhelpfully does not include upgrade instructions in the upgrade
email, so...
2020-11-03 10:19:04 +01:00
Régis Behmo
58ec1b9d79 v10.4.0 (2020-10-30)
- [Improvement] Mount config and setting files in read-only mode
- 💥[Improvement] Enable unit completion tracking by default.
- [Bugfix] Run `apt update` before `apt install` when installing deps in
the openedx Dockerfile
2020-10-30 18:20:19 +01:00
Régis Behmo
efee5b55cb Mount config and setting files in read-only mode
This is part of the process to get rid of as many writable mounts as
possible.
2020-10-25 18:44:54 +01:00
Régis Behmo
d8a4b26884 Enable unit completion tracking by default
See discussion:
https://discuss.overhang.io/t/course-not-starting/962/12
2020-10-22 15:24:40 +02:00
Rafael Sousa
dd0ab446a3 run apt update before installing deps 2020-10-19 17:51:13 +02:00
Régis Behmo
ed14fb1746 v10.3.1 (2020-10-16)
- [Improvement] Make all commands considerably faster
- [Improvement] Make it easier to override Mongodb connection parameters
- [Bugfix] Add support for .woff and .woff2 font files in themes (thanks @mrtndwrd!)
2020-10-16 12:45:40 +02:00
Régis Behmo
f32eb08940 Make it easier to override mongodb connection parameters
Pymongo clients can connect to Mongodb with many parameters
(https://pymongo.readthedocs.io/en/stable/api/pymongo/mongo_client.html#pymongo.mongo_client.MongoClient)
but we don't want to create a new Tutor setting for every one of them.
This makes it easy for users to override Mongodb connection parameters,
simply by modifying the `mongodb_parameters` object.
In particular, this makes it easy to add "replicaSet" connection
parameters.
See https://github.com/overhangio/tutor/pull/372
2020-10-15 14:35:41 +02:00
Régis Behmo
e17eede179 Add changelog entry for woff/woff2 change
See https://github.com/overhangio/tutor/pull/373
2020-10-15 14:29:27 +02:00
Régis Behmo
dac7da4bae v10.3.0 (2020-10-13)
- 💥[Improvement] Simplify CORS configuration
2020-10-13 14:57:05 +02:00
Régis Behmo
2f7742b099 Simplify CORS configuration
CORS headers are no longer handled by nginx, but directly by the LMS.
This makes it possible by 3rd-party apps to easily add domain names to
the CORS whitelist, even when they are not subdomains of the LMS.
2020-10-13 13:34:12 +02:00
Régis Behmo
9f94a12c7d v10.2.4 (2020-10-06)
- [Bugfix] Fix Apache proxy configuration when HTTPS is enabled (#370)
2020-10-06 12:01:25 +02:00
Régis Behmo
9291b2e1b9 Fix Apache proxy configuration when HTTPS is enabled
Close #370
2020-10-06 12:00:38 +02:00
Régis Behmo
e8e44f50b0 v10.2.3 (2020-10-02)
- [Feature] Add ``images printtag`` command
- [Improvement] Make it possible to override individual contact email
addresses with plugins
- [Bugfix] Replace "no-reply@LMS_BASE" email address by regular contact
email address
- [Bugfix] Disable learner records globally by default
- [Improvement] Upgrade to the latest release of MySQL 5.6
- [Improvement] Non-plugin settings added by "set" directives are now
automatically removed when the plugin is disabled (#241)
2020-10-02 14:18:52 +02:00
Régis Behmo
ccee481dd0 Add images printtag command 2020-10-02 00:25:03 +02:00
Régis Behmo
c8434d500f Fix no-reply email address in lms/cms 2020-10-01 13:41:28 +02:00
Régis Behmo
371b2b5c2d Disable learner records globally by default
Previously, platform administrators had to define a site configuration
to disable this feature.

See https://discuss.overhang.io/t/account-settings-is-not-working/967/3
2020-10-01 13:39:45 +02:00
Régis Behmo
718a6a5cf9 Upgrade to the latest release of MySQL 5.6
Hopefully, this should resolve some initialisation issues.
2020-09-23 15:33:22 +02:00
Régis Behmo
f5f501dad0 Remove "set" config entries on disabling plugin
Close #241
2020-09-18 13:21:15 +02:00
Régis Behmo
f0a3cc6072 v10.2.2 (2020-09-05)
- [Improvement] Add CORS basic configuration to LMS for subdomains of
the LMS
- [Feature] Add support for `images build --add-host` option (thanks
@grinderz!)
- [Bugfix] Fix podman compatibility by replacing `docker-compose rm`
command by `docker-compose stop` when stopping containers
- [Improvement] Improve plugin data deletion
- [Improvement] Introduce the `OPENEDX_COMMON_VERSION` setting
- [Bugfix] Make it possible to run init jobs without starting the entire
platform
- [Improvement] Reduce "openedx" Docker image size with static asset
de-duplication
2020-09-05 11:35:26 +02:00
Régis Behmo
055c3cad3f Add CORS config to LMS
By default, all subdomains of the LMS are allowed.

For reference:
https://enable-cors.org/server_nginx.html
https://stackoverflow.com/questions/54313216/nginx-config-to-enable-cors-with-origin-matching
2020-09-04 22:51:40 +02:00
Régis Behmo
b1c7dab03e Add support for images build --add-host
This was originally proposed here by @grinderz:
https://github.com/overhangio/tutor/pull/355
2020-09-04 12:35:44 +02:00
Régis Behmo
e9585bdc80 Switch to docker-compose stop for stopping containers
I do not understand the reason why we should use "rm" instead of "stop".

This fixes podman compatibility:
https://discuss.overhang.io/t/tutor-with-podman-not-working/905
2020-09-04 12:07:44 +02:00
Régis Behmo
fd50f3c384 Improve plugin data deletion 2020-09-04 12:03:50 +02:00
Régis Behmo
ea3839eb74 Introduce the OPENEDX_COMMON_VERSION setting 2020-09-03 17:33:25 +02:00
Régis Behmo
39d146d477 Make it possible to run init without starting the full platform
This is done by explicitely listing job dependencies. Unfortunately,
it's not yet possible to move `init` before `start` in `quickstart`,
because some services, such as discovery, depend on the LMS, which takes
a few seconds to boot up. Actually, discovery also depends on ngins, as
it points to "local.overhang.io" when referring to the lms.
2020-09-03 09:00:49 +02:00
Régis Behmo
30cd29e7ef Reduce openedx image size with static asset deduplication
This is inspired from FUN's openedx-docker project: static files are
de-duplicated with rdfind to reduce the uncompressed image size by
320Mb.
2020-09-01 19:15:58 +02:00
Régis Behmo
190f35d4d0 v10.2.1 (2020-08-27)
- [Bugfix] Upgrade all services to open-release/juniper.3
- [Bugfix] Fix upload of video transcripts to S3
- [Improvement] Memorize whether the user is running a production
platform during interactive configuration
2020-08-27 18:33:22 +02:00
Régis Behmo
0b4619ff90 Upgrade to juniper.3
Close #365.
2020-08-27 17:48:07 +02:00
Régis Behmo
e60a99c6af Fix upload of video transcripts to S3
See
https://discuss.overhang.io/t/video-transcript-upload-generates-500-error/836
2020-08-19 10:08:27 +02:00
Régis Behmo
bf0c5e3ed9 Memorize whether the user is running a production platform 2020-08-19 07:54:28 +02:00
Régis Behmo
2ac6f8e8dd v10.2.0 (2020-08-16)
- [Bugfix] Fix incorrect loading of some resources from localhost:18000
in development
- [Bugfix] Fix Samesite=None Secure=False cookie error for users
accessing the LMS with the latest release of Google Chrome
- [Security] Apply javascript security patch ([pull
request](https://github.com/edx/edx-platform/pull/24762))
- [Bugfix] Fix "FileError" on Scorm package upload in Scorm XBlock
- 💥[Improvement] Serve openedx static assets with
[whitenoise](http://whitenoise.evans.io/en/stable/) instead of nginx.
This removes the `k8s-deployments-nginx-init-containers` patch. Plugins
are encouraged to implement static asset serving with Whitenoise as
well.
- [Bugfix] Fix dependency on mysql service when mysql is not activated
- [Improvement] Improve openedx Docker image build time and size with
multi-stage build
- 💥[Feature] Get rid of outdated sysadmin dashboard in LMS at /sysadmin
2020-08-16 12:02:39 +02:00
Régis Behmo
699c589038 Mark sysadmin dashboard changelog entry as breaking 2020-08-16 12:02:35 +02:00
Régis Behmo
0d0cb0e71e Fix incorrect loading of some resources from localhost:18000
In the LMS, some resources where loaded from localhost:18000. For
instance: http://localhost:18000/static/images/logo.png

This was due to the fact that the LMS_BASE, LMS_ROOT_URL and thus
SITE_NAME settings are overwritten by the devstack settings, so we need
to define them again in development.
2020-08-16 11:56:37 +02:00
Régis Behmo
9a6439b08c Fix Samesite=None Secure=False cookie errors in Chrome
Recent releases of Chrome refuse to set csrf and session cookies for
which secure=False samesite=None. The "secure" attribute is not set by
the SameSite middleware in v0.5.1. It was introduced in v0.6.0.

Instead, the "secure" attribute is set on the sessions cookie by the
openedx.core.djangoapps.safe_sessions.middleware.SafeSessionMiddleware
middleware. For the csrf cookie, the "secure" attribute is set by
Django.

We could certainly get rid of the SafeSessionMiddleware by upgrading the
django-cookie-samesite dependency to v0.6.0. Instead, we need to define
environment-specific settings manually.

See:
https://github.com/edx/edx-platform/pull/23671
https://github.com/edx/edx-platform/pull/24593
https://discuss.overhang.io/t/users-cannot-login-csrf-cookie-not-set/815
https://discuss.openedx.org/t/lti-xblock-and-samesite/759/3
https://blog.heroku.com/chrome-changes-samesite-cookie
https://docs.djangoproject.com/en/2.2/ref/settings/#csrf-cookie-secure
https://github.com/jotes/django-cookies-samesite/issues/23
2020-08-16 11:39:00 +02:00
Régis Behmo
1d5bc05328 Apply javascript security patch
See pull request: https://github.com/edx/edx-platform/pull/24762
2020-08-15 19:18:45 +02:00
Régis Behmo
a161dbf843 Fix "FileError" on Scorm package upload in Scorm XBlock
See https://github.com/overhangio/openedx-scorm-xblock/issues/3
2020-08-15 17:24:11 +02:00
Régis Behmo
62d7fd00b9 Serve static assets with whitenoise instead of nginx
This drastically simplifies volume management, as it is no longer
necessary to manually copy static assets from the docker image to the
bind-mounted volume.

This deprecates the "k8s-deployments-nginx-init-containers" patch, as we
no longer need to init the nginx container. Plugins are encouraged to
start using whitenoise as well for serving static assets.

TODO:
- test media serving: DOES NOT WORK. Whitenoise was designed to serve a
fixed list of static files. Godammit.
- compare performances
2020-08-04 09:39:53 +02:00
Régis Behmo
895be87bca Fix dependency on mysql service when mysql is not activated
See https://github.com/overhangio/tutor/pull/359
2020-08-02 16:46:20 +02:00
Régis Behmo
398e1b96e8 Switch to multi-stage build for openedx image
This reduces the size of the final image from 3.25Gb to 2.8Gb. Also, it
should be faster to rebuild the image in most cases. For instance, we
will not have to re-install nodejs requirements after part of the
edx-platform repo was modified.
2020-07-26 00:51:26 +02:00
Régis Behmo
c46cab3df9 Revert "Enable LMS sysadmin dashboard by default"
This reverts commit 90c5842c1c.

Turns out that the sysadmin dashboard is slated for deprecation. It is
currently impossible to download the list of users in csv (see
https://discuss.overhang.io/t/tutor-10-0-11-sysadmin-feature-generates-500-error/776/4).
We should not attempt to resolve this issue.
2020-07-24 08:31:42 +02:00
Régis Behmo
ddee6f4d87 v10.1.0 (2020-07-23)
- [Security] Apply edx-platform upstream xss security fixes ([pull
request](https://github.com/edx/edx-platform/pull/24568))
- 💥[Feature] Make it possible to override the docker registry for just
a few services by setting `DOCKER_IMAGE_SERVICENAME` values.
2020-07-23 16:19:26 +02:00
Régis Behmo
8e2a06dc6f Apply edx-platform upstream xss security fixes 2020-07-23 16:18:40 +02:00
Régis Behmo
bbd92223ee Make it possible to override the docker registry for individual services
Previously, it was not possible to override the docker registry for just
one or a few services. Setting the DOCKER_REGISTRY configuration
parameter would apply to all images. This was inconvenient. To resolve
this, we include the docker registry value in the DOCKER_IMAGE_*
configuration parameters. This allows users to override the docker
registry individually by defining the DOCKER_IMAGE_SERVICENAME
configuration parameter.

See https://discuss.overhang.io/t/kubernetes-ci-cd-pipeline/765/3
2020-07-21 09:16:51 +02:00
Régis Behmo
d5659cc03b v10.0.11 (2020-07-16)
- [Feature] Upgrade all repositories to open-release/juniper.2
- [Bugfix] Fix `reload-gunicorn` command
- [Feature] Enable sysadmin dashboard in LMS at /sysadmin
2020-07-16 15:33:28 +02:00
Régis Behmo
ee0f6de499 Add changelog entry for reload-gunicorn fix
See https://github.com/overhangio/tutor/pull/354
2020-07-16 11:58:13 +02:00
Régis Behmo
ef6b1c3433 Upgrade to open-release/juniper.2 2020-07-16 11:57:35 +02:00
Régis Behmo
90c5842c1c Enable LMS sysadmin dashboard by default
The dashboard is available at /sysadmin. It's a CRUD interface for
managing users and courses.

Enabling this interface required that the DATA_DIR setting was not a
string, but a Path object.

Close #353.
2020-07-16 10:49:57 +02:00
Régis Behmo
66ab851848 v10.0.10 (2020-07-01)
- [Bugfix] Fix pycontracts installation error when building openedx
Docker image
- [Bugfix] Fix access to dicussion forum in development mode
2020-07-01 18:27:16 +02:00
Régis Behmo
e1e6a00c0f Fix pycontracts error in openedx docker image build
See
https://discuss.overhang.io/t/tutor-images-build-openedx-failed/717/2
2020-07-01 18:25:51 +02:00
Régis Behmo
a552c5af6f Fix access to dicussion forum in development mode
The devstack settings from edx-platform set the forum url to
http://edx.devstack.forum:4567 so we need to override this value.
2020-07-01 11:04:14 +02:00
Régis Behmo
521b7e8098 v10.0.9 (2020-07-07)
- [Bugfix] Share grade download settings between the LMS and the CMS
2020-07-01 10:10:38 +02:00
Régis Behmo
169a96c44f Share grade download settings between the LMS and the CMS
The GRADES_DOWNLOAD setting is shared between the LMS and the CMS, so
its definition is moved to the common settings file.

This is to address part of this issue:
https://discuss.overhang.io/t/grades-file-not-generating-juniper-version/704
An upgrade to the minio plugin is also required.
2020-07-01 10:06:53 +02:00
Régis Behmo
a9907d7eb3 v10.0.8 (2020-06-23)
- [Bugfix] Fix android user creation during init
- [Bugfix] Fix undefined settings in k8s scripts, such as `createuser`
2020-06-23 19:27:14 +02:00
Régis Behmo
f28ecca498 Fix android user creation during init
Init was crashing with error:

    django.db.utils.IntegrityError: (1062, "Duplicate entry 'android' for key 'client_id'")

See
https://discuss.overhang.io/t/reset-password-email-sent-but-activation-email-dont/690/4
2020-06-23 19:25:56 +02:00
Régis Behmo
b328e0f7d9 Fix undefined settings in k8s scripts
In k8s, creating a user is an interactive command, so it needs to run in
exec. Thus, the DJANGO_SETTINGS_MODULE needs to be defined for this
command.

Close #344
2020-06-23 19:11:35 +02:00
Régis Behmo
2fc28bef79 v10.0.7 (2020-06-22)
- [Bugfix] Fix TypeError on viewing xblock
- [Bugfix] Fix authentication in Android mobile application
2020-06-22 16:31:31 +02:00
Régis Behmo
d8676978f1 Fix TypeError due to None LEARNING_MICROFRONTEND_URL setting value
We observed the following error in production:

    lms_1             | 2020-06-22 14:03:10,220 ERROR 10 [django.request]
    [user 4] log.py:228 - Internal Server Error:
    /xblock/block-v1:edX+DemoX+Demo_Course+type@html+block@6bcccc2d7343416e9e03
    fd7325b2f232
    lms_1             | Traceback (most recent call last):
    lms_1             |   File
    "/openedx/venv/lib/python3.5/site-packages/django/core/handlers/exception.py",
    line 34, in inner
    lms_1             |     response = get_response(request)
    lms_1             |   File
    "/openedx/venv/lib/python3.5/site-packages/django/core/handlers/base.py",
    line 115, in _get_response
    lms_1             |     response =
    self.process_exception_by_middleware(e, request)
    lms_1             |   File
    "/openedx/venv/lib/python3.5/site-packages/django/core/handlers/base.py",
    line 113, in _get_response
    lms_1             |     response = wrapped_callback(request,
    *callback_args, **callback_kwargs)
    lms_1             |   File
    "/openedx/venv/lib/python3.5/site-packages/django/views/decorators/http.py",
    line 40, in inner
    lms_1             |     return func(request, *args, **kwargs)
    lms_1             |   File
    "/openedx/edx-platform/common/djangoapps/util/views.py", line 66, in
    inner
    lms_1             |     response = view_func(request, *args, **kwargs)
    lms_1             |   File
    "/openedx/venv/lib/python3.5/site-packages/django/views/decorators/clickjacking.py",
    line 50, in wrapped_view
    lms_1             |     resp = view_func(*args, **kwargs)
    lms_1             |   File
    "/openedx/venv/lib/python3.5/site-packages/django/utils/decorators.py",
    line 142, in _wrapped_view
    lms_1             |     response = view_func(request, *args, **kwargs)
    lms_1             |   File
    "/openedx/edx-platform/lms/djangoapps/courseware/views/views.py", line
    1683, in render_xblock
    lms_1             |     'is_learning_mfe':
    request.META.get('HTTP_REFERER',
    '').startswith(settings.LEARNING_MICROFRONTEND_URL),
    lms_1             | TypeError: startswith first arg must be str or a
    tuple of str, not NoneType
2020-06-22 16:18:21 +02:00
Régis Behmo
c2e30a6855 Fix authentication in Android mobile application
Login on Android mobile app was failing with "check your username and
password".
2020-06-22 16:01:46 +02:00
Régis Behmo
92c4055b36 v10.0.6 (2020-06-22)
- [Bugfix] Fix unsent activation emails and other asynchronous tasks
2020-06-22 12:19:57 +02:00
Régis Behmo
40a76330f6 Fix unsent activation emails and other asynchronous tasks
Half of the tasks from edx.lms.core.default celery queue were being
processed by the CMS worker. Unfortunately, this CMS worker crashes on
some of those tasks. For instance, activation emails complain of a
missing "django_markup" template tag library because "xss_utils" is not
part of the installed app in the CMS.

The problem is that we need this edx.lms.core.default queue to be part
of the CELERY_QUEUES in the cms in order to send tasks from the CMS to
the LMS. The trick to resolve this situation is to ask the CMS celery
worker to not process the tasks from this queue.

To debug this issue, run in the LMS:

    from student.tasks import send_activation_email
    send_activation_email("{}")

Then watch the logs of the lms and cms workers. If the CMS workers picks
up this task (50% of the time prior to this change) then we have an
issue.

See:
https://discuss.overhang.io/t/reset-password-email-sent-but-activation-email-dont/690
2020-06-22 12:11:15 +02:00
Régis Behmo
cdf6b7e315 v10.0.5 (2020-06-21)
- [Security] Apply edx-platform upstream xss security fixes ([pull
request](https://github.com/edx/edx-platform/pull/24258))
2020-06-21 13:46:19 +02:00
Régis Behmo
1773e2a347 Apply edx-platform upstream xss security fixes 2020-06-21 11:04:46 +02:00
Régis Behmo
77a0a6f60d v10.0.4 (2020-06-19)
- [Bugfix] Fix broken `tutor ui` command in binary
2020-06-19 16:37:58 +02:00
Régis Behmo
2b0ed7daac Fix broken ui command in tutor binary
"tutor ui" was failing miserably, printing a lot of garbled characters
in the shell. In fact, a FileNotFound error was being raised and
automatically caught by the ui command. When removing the catch all,
this was the error that was raised:

    FileNotFoundError: [Errno 2] No such file or directory:
    '/tmp/_MEIimsqmq/wcwidth/version.json'     │

This is resolved on SO:
https://stackoverflow.com/questions/62155242/pyinstaller-cant-find-wcwidth-version-json-when-running-executable
2020-06-19 16:33:58 +02:00
Régis Behmo
001da4e24e v10.0.3 (2020-06-19)
- [Bugfix] Fix error on android user creation
2020-06-19 12:53:18 +02:00
Régis Behmo
2f8efbc656 Revert "Fix DoesNotExist error on android credentials creation"
This reverts commit a88d9cfc19.
2020-06-19 11:40:46 +02:00
Régis Behmo
d9aab8b314 v10.0.1 (2020-06-17)
- [Bugfix] Fix crash when viewing problem in LMS
- [Bugfix] Fix missing webpack-stats.json in openedx Docker image
2020-06-17 21:51:52 +02:00
Régis Behmo
baca27bfbd Fix crash when viewing problem in LMS
This was due to incorrectly loading the coursewarehistoryextended in the
installed applications. Also, the database router in charge of routing
requests to the student_history_module database must be disabled.
2020-06-17 11:10:37 +02:00
Régis Behmo
325c1037c0 Fix missing webpack-stats.json in openedx Docker image
In CI, the webpack-stats.json file sometimes contains just:

    {"status":"compiling"}

This was due to the fact that the `subprocess.call(...)` command in
openedx-assets did not check whether the command was killed -- for lack
of memory for instance. This is resolved by replacing "call(...)" by
"check_call(...)".
2020-06-16 06:22:12 +02:00
Régis Behmo
24035ac1fe v10.0.1 (2020-06-15)
- [Bugfix] Fix KeyError when running ``local quickstart`` for the first
time
2020-06-15 17:59:27 +02:00
Régis Behmo
06fe19fcf2 Fix KeyError when running `local quickstart` for the first time
This was due to incorrect parsing of the version number.
2020-06-15 17:57:14 +02:00
Régis Behmo
4d6de0138a v10.0.0 Upgrade to Juniper (2020-06-15)
Here, we upgrade the Open edX platform from Ironwood to Juniper. This
upgrade does not come with many feature changes, but there are many
technical improvements under the hood:

- Upgrade from Python 2.7 to 3.5
- Upgrade from Mongodb v3.2 to v3.6
- Upgrade Ruby to 2.5.7

We took the opportunity to completely rething the way locally running
platforms should be accessed for testing purposes. It is no longer
possible to access a running platform from http://localhost and
http://studio.localhost. Instead, users should access
http://local.overhang.io and https://studio.local.overhang.io. This
drastically simplifies internal communication between Docker containers.

To upgrade, users should simply run:

    tutor local quickstart

For Kubernetes platform, the upgrade process is outlined when running:

    tutor k8s upgrade --from=ironwood
2020-06-15 10:19:07 +02:00
Régis Behmo
29c6741bba Fix nginx resolver address 2020-06-14 15:47:37 +02:00
Régis Behmo
872a33414a Add --limit=myplugin option to init commands
This makes it possible to run "init" for some services only, which is
useful during debugging.
2020-06-02 14:30:06 +02:00
Régis Behmo
a6c9bba3bf v3.12.6 (2020-06-01)
- [Improvement] Add `dig`, `ping` utilities to openedx-dev Docker image
- [Bugfix] Resolve "Can't connect to MySQL server" on init
- [Improvement] Make it possible to customize the MySQL root username,
for connecting to external MySQL databases
2020-06-01 17:20:35 +02:00
Régis Behmo
092c3955c1 Add dig, ping utilities to openedx-dev docker image 2020-06-01 17:19:41 +02:00
Régis Behmo
47aaf63a59 Attempt to resolve "Can't connect to MySQL server" on init
Close #334 (hopefully)
2020-06-01 17:14:22 +02:00
Régis Behmo
3831e2e53d Make the mysql root username a configuration parameter
This allows users who connect to an external database to customise the
root username. Indeed, quite often the "root" username is reserved.

Close #328
2020-06-01 17:11:10 +02:00
Régis Behmo
0711dd462b v3.12.5 (2020-05-20)
- [Improvement] Upgrade Android app to v2.21.1 and enable many features,
such as downloading videos to SD card. Thanks for the help @ejklock!
- [Bugfix] Fix Android app crash when accessing course
2020-05-20 17:20:06 +02:00
Régis Behmo
a92ff42348 Fix and upgrade Android app to v2.21.1
Also, we introduce tons of cool features, including downloading videos
to SD card.
2020-05-20 17:19:13 +02:00
Régis Behmo
cce02dcdf6 v3.12.4 (2020-05-18)
- [Improvement] Add ability to rescore SCORM units
- [Bugfix] Fix scoring of graded SCORM units
- [Improvement] Increase maximum uploaded file size in the CMS from 10
to 100 Mb.
2020-05-18 11:35:31 +02:00
Régis Behmo
8035b7dfc6 Upgrade scorm to 9.2.0
This adds a bugfix (float graded submissions) and a new feature
(assignment rescoring).
2020-05-18 11:33:58 +02:00
Régis Behmo
b30617581d Increase maximum uploaded file size in the CMS
See discussion: https://discuss.overhang.io/t/increase-upload-size/556
2020-05-13 06:54:03 +02:00
Régis Behmo
95271092c5 v3.12.3 (2020-05-05)
- [Security] Apply most recent edx-platform [security
patches](https://discuss.openedx.org/t/security-patch-for-edit-chapter-xss-lint-issues/2030)
2020-05-05 18:58:01 +02:00
Régis Behmo
be38f70f5d Apply most recent edx-platform security patches
See this conversation:
https://discuss.openedx.org/t/security-patch-for-edit-chapter-xss-lint-issues/2030
2020-05-05 17:04:34 +02:00
Régis Behmo
c18ab3e7c6 v3.12.2 (2020-04-29)
- [Bugfix] Fix oauth2 authentication with 3rd-party services, such as discovery
- [Bugfix] Upgrade scorm xblock to solve caching issue
2020-04-29 16:15:10 +02:00
Régis Behmo
aef0ede0ec Fix oauth with services such as discovery
See https://discuss.overhang.io/t/problem-occured-during-checkout-please-contact-support/491/31
2020-04-29 15:53:01 +02:00
Régis Behmo
9043450de8 Upgrade scorm xblock to solve caching issue
See https://github.com/overhangio/openedx-scorm-xblock/issues/1
2020-04-29 11:23:27 +02:00
Régis Behmo
be6bc76b2c v3.12.1 (2020-04-27)
- [Improvement] Increase max upload allowed size to 250Mb in CMS
2020-04-27 00:27:17 +02:00
Régis Behmo
1815ab138e Increase allowed upload size for studio content
This is particularly useful for courses with large SCORM content.
2020-04-27 00:26:39 +02:00
Régis Behmo
47aad68a5c v3.12.0 (2020-04-26)
- 💥[Improvement] Do not deploy an ingress or SSL/TLS certificate issuer
ressource by default in Kubernetes
- [Improvement] Fix tls certificate generation in k8s
- 💥[Improvement] Radically change the way jobs are run: we no longer
"exec", but instead run a dedicated container.
- 💥[Improvement] Upgrade k8s certificate issuer to
cert-manager.io/v1alpha2
- [Feature] Add SCORM XBlock to default openedx docker image
2020-04-26 01:05:48 +02:00
Régis Behmo
a52884a311 Remove ingress/issuer from default k8s deployment
There are too many different ways to deploy an Ingress resource and to
generate SSL/TLS certificates: it's too much responsibility to make that
decision for the end user.
2020-04-25 23:12:42 +02:00
Régis Behmo
bce6432d85 Improve job running in local and k8s
Running jobs was previously done with "exec". This was because it
allowed us to avoid copying too much container specification information
from the docker-compose/deployments files to the jobs files. However,
this was limiting:

- In order to run a job, the corresponding container had to be running.
This was particularly painful in Kubernetes, where containers are
crashing as long as migrations are not correctly run.
- Containers in which we need to run jobs needed to be present in the
docker-compose/deployments files. This is unnecessary, for example when
mysql is disabled, or in the case of the certbot container.

Now, we create dedicated jobs files, both for local and k8s deployment.
This introduces a little redundancy, but not too much. Note that
dependent containers are not listed in the docker-compose.jobs.yml file,
so an actual platform is still supposed to be running when we launch the
jobs.

This also introduces a subtle change: now, jobs go through the container
entrypoint prior to running. This is probably a good thing, as it will
avoid forgetting about incorrect environment variables.

In k8s, we find ourselves interacting way too much with the kubectl
utility. Parsing output from the CLI is a pain. So we need to switch to
the native kubernetes client library.
2020-04-25 23:12:42 +02:00
Régis Behmo
091e45fe63 Fix tls certificate generation in k8s
The "Certificate" objects are no longer required. As a consequence, the
"k8s-ingress-certificates" has become useless and should be removed from
plugins.
2020-04-25 14:40:48 +02:00
Régis Behmo
e4ca99b237 Upgrade cert-manager installation in k8s
Note that the spec has changed for v1alpha2.
2020-04-25 14:40:48 +02:00
Régis Behmo
49c71f8af2 Add scorm xblock to the openedx docker image 2020-04-25 14:20:13 +02:00
Régis Behmo
0960449405 v3.11.12 (2020-04-16)
- [Feature] Make it easy to add custom translation strings to the
openedx Docker image
- [Improvement] Make it possible to rely on a different npm registry for
faster image building
2020-04-16 19:31:36 +02:00
Régis Behmo
33ab52bfeb Make it easy to add custom translation strings to edx-platform
Users can now add custom translation strings to a locale folder at build
time, very much in the same way as custom themes or requirements. This
is quite convenient, although is does require quite a bit of time to
rebuild the docker images.
2020-04-16 19:30:08 +02:00
Régis Behmo
76d1078382 Make it possible to pull nodejs packages from custom npm registry
During an incident at npmjs.org it was extremely difficult to pull
nodejs packages -- so we made it possible to pull from a custom
registry, deployed for instance with Verdaccio (https://verdaccio.org/).
2020-04-16 19:26:02 +02:00
Régis Behmo
bdc0cb04e2 v3.11.11 (2020-04-15)
- [Bugfix] Make sure all emails (including "password reset") are
properly saved to a local file in development mode (#315)
- [Improvement] Add `openedx-development-settings` patch to patch the
LMS and the CMS simultaneously in development
- [Bugfix] Fix missing celery tasks in the CMS
2020-04-15 23:53:05 +02:00
Régis Behmo
724c2c84da Make sure all emails are stored to a tmp file in development
In development, emails sent from edx-platform were using the
"file_email" channel from edx-ace ("edX's automated communication
engine"). This channel was failing because it tries to write to a file
located in the /edx folder, which does not exist in tutor containers. To
fix this, we configure edx-ace to rely on the django email backend,
which itself is configured to send emails to a file in development. It
turns out that this backend was also configured to store emails in a
file located in the /edx folder, so we had to add the standard
EMAIL_FILE_PATH django setting to our development settings.

It was easier to reconfigure the django file email backend than the
edx-ace file_email channel because the output path of the latter cannot
be modified by a setting.

Note that this causes all emails to be stored in local files instead of
being sent to actual recipients. This is the default behaviour in Open
edX, and indeed in most default django apps (in development). This is a
good thing! If, for some reason, developers would like to try out email
sending during development, they should modify the EMAIL_BACKEND
setting and set it to 'django.core.mail.backends.smtp.EmailBackend'.
This is quite easy to achieve with the help of a plugin:

    name: sendemailsindev
    version: 0.1.0
    patches:
      openedx-development-settings: |
          # actually send emails in dev
          EMAIL_BACKEND = "django.core.mail.backends.smtp.EmailBackend"

Close #315
2020-04-15 23:52:28 +02:00
Régis Behmo
072c3a1a15 Add openedx-development-settings patch 2020-04-15 23:52:18 +02:00
Régis Behmo
54d9fe20e7 Fix missing celery tasks in the CMS
This should spare us quite a few error messages in the CMS worker.
2020-04-14 19:49:18 +02:00
Régis Behmo
68ff61a421 v3.11.10 (2020-04-14)
- [Bugfix] Fix updates to the course structure in the LMS (#302)
2020-04-14 19:24:27 +02:00