6
0
mirror of https://github.com/ChristianLight/tutor.git synced 2024-12-13 14:43:03 +00:00
Commit Graph

1306 Commits

Author SHA1 Message Date
Régis Behmo
db818ab4a1 fix: do not save the full config on "plugins en/disable"
This was saving the full configuration to config.yml, resulting in many
incorrect configuration values...
2021-12-20 21:19:10 +01:00
Régis Behmo
d82546adde fix: breaking "config save"
"config save" was not loading the full configuration prior to saving the
environment.
2021-12-20 21:19:10 +01:00
Régis Behmo
c40e682f5d refactor: clarify configuration management
Previously, configuration management was very confusing because we kept mixing
"base" and "defaults" configuration:

- It was difficult to make the difference between core settings that were
  necessary (e.g: passwords) as opposed to others that could simply be
  defaulted to.
- The order of settings in config.yml mattered: config entries that depended on
  other needed to be defined later. As a consequence, Tutor was not compatible
  with Python 3.5, where dict entries are not sorted.
2021-12-20 21:19:10 +01:00
Régis Behmo
b18c9dc4f8 fix: lint unused arguments in code base 2021-12-20 21:19:10 +01:00
Régis Behmo
28edf106c4 chore: drop python 3.5 compatibility
Python 3.5 has reached end of life in September 3.5. Anyway, Tutor was not
compatible because some dev dependencies, such as astroid 2.8.3, are no longer
available in 3.5.

This means that we can now start using many python 3.6 niceties, such as
f-strings \o/
2021-12-20 21:19:10 +01:00
Régis Behmo
3d92087f3a fix: docker-compose project name in dev on nightly
Project name was incorrectly set to "tutor_dev" instead of "tutor_nightly_dev".
2021-12-20 21:19:10 +01:00
Florian Haas
9446095665 fix: Stop creating immutable resource label that breaks "tutor k8s" on Tutor version changes
Through the commonLabels directive in kustomization.yml, all resources
get a label named "app.kubernetes.io/version", which is being set to
the Tutor version at the time of initial deployment.

When the user then subsequently progresses to a new Tutor version,
Kubernetes attempts to update this label — but for Deployment,
ReplicaSet, and DaemonSet resources, this is no longer allowed as of
https://github.com/kubernetes/kubernetes/issues/50808. This causes
"tutor k8s start" (at the "kubectl apply --kustomize" step) to break
with errors such as:

Deployment.apps "redis" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{"app.kubernetes.io/instance":"openedx-JIONBLbtByCGUYgHgr4tDWu1", "app.kubernetes.io/managed-by":"tutor", "app.kubernetes.io/name":"redis", "app.kubernetes.io/part-of":"openedx", "app.kubernetes.io/version":"12.1.7"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable

Simply removing the app.kubernetes.io/version label from
kustomization.yml will permanently fix this issue for newly created
Kubernetes deployments, which will "survive" any future Tutor version
changes thereafter.

However, *existing* production Open edX deployments will need to throw
the affected Deployments away, and re-create them.

Also, add the Tutor version as a resource annotation instead, using
the commonAnnotations directive.

See also:
https://github.com/kubernetes/client-go/issues/508
https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/commonlabels/
https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/commonannotations/

Fixes #531.
2021-12-20 21:19:10 +01:00
Régis Behmo
0917da59bf fix: don't attempt to security patch edx-platform 2021-12-20 21:19:10 +01:00
Régis Behmo
dc5307ad75 fix: get rid of useless redis folder in permission setting
Adding these volumes was a mistake.
2021-12-20 21:19:10 +01:00
Régis Behmo
c117a300aa fix: disable forum feature by default 2021-12-20 21:19:10 +01:00
Régis Behmo
0c1d4ebf51 fix: https test 2021-12-20 21:19:10 +01:00
Régis Behmo
7b5ec22e0c fix: enable plugins to implement the "caddyfile" patch
When nginx was removed in favour of caddy, we decided that plugin
implementations of the "caddyfile" patch should make use of the "port" local
variable. However, local variables are not available from inside plugin
patches, which are rendered outside of the context of the parent templates.

For a more extensive description of the problem, see:
https://github.com/overhangio/tutor-mfe/pull/23#issuecomment-964016190

We still want to make it easy for developers to decide what should the port be
for caddy hosts. To do so, we make use of environment variables that are passed
at runtime to the caddy container.

Thus, a regular plugin patch should look like this:

    {{ PLUGIN_HOST }}{$default_site_port} {
        import proxy "myplugin:8000"
    }
2021-12-20 21:19:10 +01:00
Régis Behmo
b8ab829c11 feat: move all forum-related code to a dedicated plugin
Forum is an optional feature, and as such it deserves its own plugin. Starting
from Maple, users will be able to install the forum from
https://github.com/overhangio/tutor-forum/

Close #450.
2021-12-20 21:19:10 +01:00
Régis Behmo
6f7457f6b3 doc: remove now irrelevant comment 2021-12-20 21:19:10 +01:00
Régis Behmo
9964315abe chore: get rid of tutor-openedx
In the past, tutor was installed with "pip install tutor-openedx". For
some time (since v12.0.2), "tutor" was installed as a dependency of
"tutor-openedx". Now is the time to get rid of that old package.
The standard way of installing tutor is now with "pip install tutor".
2021-12-20 21:19:10 +01:00
Régis Behmo
0658be86b8 fix: always run Caddy on Kubernetes
Caddy should always be running, even when ENABLE_WEB_PROXY is false.
It's the service that should not always be running.
2021-12-20 21:19:10 +01:00
Régis Behmo
ce5310db06 fix: permissions image name 2021-12-20 21:19:10 +01:00
Régis Behmo
01b58d9d75 feat: run all services as unprivileged containers
With this change, containers are no longer run as "root" but as unprivileged
users. This is necessary in some environments, notably some Kubernetes
clusters.

To make this possible, we need to manually fix bind-mounted volumes in
docker-compose. This is pretty much equivalent to the behaviour in Kubernetes,
where permissions are fixed at runtime if the volume owner is incorrect. Thus,
we have a consistent behaviour between docker-compose and Kubernetes.

We achieve this by bind-mounting some repos inside "*-permissions" services.
These services run as root user on docker-compose and will fix the required
permissions, as per build/permissions/setowner.sh These services simply do not
run on Kubernetes, where we don't rely on bind-mounted volumes. There, we make
use of Kubernete's built-in volume ownership feature.

With this change, we get rid of the "openedx-dev" Docker image, in the sense
that it no longer has its own Dockerfile. Instead, the dev image is now simply
a different target in the multi-layer openedx Docker image. This makes it much
faster to build the openedx-dev image.

Because we declare the APP_USER_ID in the dev/docker-compose.yml file, we need
to pass the user ID from the host there. The only way to achieve that is with a
tutor config variable. The downside of this approach is that the
dev/docker-compose.yml file is no longer portable from one machine to the next.
We consider that this is not such a big issue, as it affects the development
environment only.

We take this opportunity to replace the base image of the "forum" image. There
is now no need to re-install ruby inside the image. The total image size is
only decreased by 10%, but re-building the image is faster.

In order to run the smtp service as non-root, we switch from namshi/smtp to
devture/exim-relay. This change should be backward-compatible.

Note that the nginx container remains privileged. We could switch to
nginxinc/nginx-unprivileged, but it's probably not worth the effort, as we are
considering to get rid of the nginx container altogether.

Close #323.
2021-12-20 21:19:10 +01:00
Régis Behmo
231bbbfe99 feat: get rid of the nginx container and services
Nginx and Caddy performed duplicate tasks. It was decided to get rid of
the nginx container, for simplification. This is a breaking change for
plugin developers. Also, applications that collect nginx logs will have
to be modified.

See:
- Corresponding TEP: https://discuss.overhang.io/t/tep-get-rid-of-the-nginx-container/2024
- the prior discussion: https://discuss.overhang.io/t/why-caddy-nginx/1952
2021-12-20 21:19:10 +01:00
Régis Behmo
670552ca5f fix: lms 500 error caused by missing LANGUAGE_COOKIE_NAME setting
See also: https://github.com/overhangio/tutor/pull/507
Upstream fix: https://github.com/edx/edx-platform/pull/29096
2021-12-20 21:19:10 +01:00
Régis Behmo
01fb62bf29 fix: running mongodb locally and on k8s 2021-12-20 21:19:10 +01:00
Régis Behmo
d960e18f28 chore: upgrade elasticsearch/mongodb/redis
Open edX master now runs elasticsearch 7.10 and mongodb 4.2. Redis also
received a minor upgrade.
2021-12-20 21:19:10 +01:00
Braden MacDonald
1f46ed39c5 fix: openedx image wasn't building on ARM64 due to missing libgeos-dev 2021-12-14 12:10:49 +01:00
Régis Behmo
e89ff49223 v12.2.0
- 💥[Improvement] Fail on incorrect image name argument in `images
build/pull/push/printtag` commands.
- [Bugfix] Remove trailing slashes in docker-compose files for
[compatibility with docker-compose v2 in
WSL](https://github.com/docker/compose/issues/8558).
- [Improvement] `settheme` now works with preview domain.
- [Feature] Allow specifying extra pip packages through config.yml.
2021-12-08 11:29:52 +01:00
Régis Behmo
79c6d122e2 fix: incorrect "from" address in course bulk emails
Upstream PR: https://github.com/edx/edx-platform/pull/29001
See discussion:
https://discuss.overhang.io/t/sending-bulk-email-triggers-smtprecipientsrefused-error/1923
Related issue:
https://github.com/openedx/build-test-release-wg/issues/102
2021-12-08 11:29:52 +01:00
Crist Ye
ddcfab2ddb docs: improve plugin defaults readability 2021-12-08 11:06:03 +01:00
Régis Behmo
c3e6f57b08 feat: fail images ... [image] commands on incorrect image arg
Previously, `tutor images build doesnotexist` was a no-op. This was
counter-intuitive -- in particular, when users have a typo in their command.

Now, the `images build/pull/push/printtag` commands fail if the corresponding
image was not found.

See discussion: https://discuss.overhang.io/t/return-something-when-tutor-images-build-contains-a-typo/2174
2021-12-02 09:34:16 +01:00
Maxim Beder
19157d94bc feat: allow to specify extra pip packages in config
Added OPENEDX_EXTRA_PIP_REQUIREMENTS setting, which allows to specify
extra pip packages that should be installed.

Moved "openedx-scorm-xblock" package from Dockerfile to the new setting
in the config.yml.
2021-11-30 11:32:40 +01:00
iamcristye@outlook.com
5604cd4bf8 Improvement: settheme now works with preview domain 2021-11-29 18:21:16 +01:00
Régis Behmo
d4fe1260c1 docs: add tutorial on using Google Mail as an SMTP server
The tutorial comes from the feedback of multiple users, including:
https://discuss.overhang.io/t/google-smtp-not-working/2143
2021-11-25 12:25:23 +01:00
Régis Behmo
23847f6441 docs: keep running "make watch" on failure 2021-11-25 12:25:23 +01:00
Sofiane Bebert
852759ae35 fix: remove trailing slashes in volume paths for docker-compose v2 compatibility
close #522
2021-11-22 14:10:28 +01:00
Régis Behmo
a4dcb9eaf2 v12.1.7 (2021-11-18)
- [Security] Timed exam security fix [29347](https://github.com/edx/edx-platform/pull/29347).
- [Feature] Add [tutor-richie](https://github.com/overhangio/tutor-richie) to the plugins that are bundled with the tutor binary.
- [Improvement] Make `tutor plugins list` print plugins sorted by name.
- [Improvement] Ignore Python plugins which cannot be loaded.
- [Bugfix] When configured with `RUN_FORUM: false`, omit forum-related [Jobs](https://kubernetes.io/docs/concepts/workloads/controllers/job/) from the manifests that `tutor k8s` generates. (#525)
2021-11-18 11:04:15 +01:00
Régis Behmo
aa931b33f4 security: applied timed exam fix 2021-11-18 10:55:50 +01:00
Florian Haas
0c3a160fce fix: Include the forum job only if RUN_FORUM is set
When running "tutor k8s" we can disable the forum deployment and
service by setting RUN_FORUM to false, but the corresponding job is
added to jobs.yml unconditionally.

Add a conditional to tutor/templates/k8s/jobs.yml so that the forum
job definition is only included when RUN_FORUM is true.

Fixes #525.
2021-11-18 10:22:54 +01:00
Régis Behmo
0a4a2bbac2 feat: add richie to bundled plugins
For more information, checkout the plugin docs:
https://github.com/overhangio/tutor-richie
2021-11-09 11:48:32 +01:00
Régis Behmo
485f47f6d0 fix: always print plugins sorted by name
`tutor plugins list` used to print plugins in random oredr. To be honest
this has always bothered me.
2021-11-08 10:40:37 +01:00
Régis Behmo
d9d08ad0f7 improvement: don't crash on unloadable plugins
In some cases, plugins declare an entry point but cannot be loaded.
This is the case when they depend on a version of tutor that is not the
one that is currently installed. This use case is very frequent when
working on multiple versions at the same time (i.e: right now, while we
are working on the Maple release). In such cases, it's best just to
ignore the plugin entirely rather than having to re-install all plugins
in the virtualenv.
2021-11-08 10:40:37 +01:00
Régis Behmo
0a2abe32dc v12.1.6 (2021-11-02)
- [Improvement] Upgrade all services to open-release/lilac.3.
- [Feature] Make it possible to override job configuration in
development: if they exist, `dev/docker-compose.jobs.yml` and
`dev/docker-compose.jobs.override.yml` will be loaded when running jobs.
- [Improvement] Faster `tutor local start` by building only necessary
images.
2021-11-02 15:55:10 +01:00
Régis Behmo
c9a08a5e18 chore: upgrade to open-release/lilac.3 2021-11-02 11:56:24 +01:00
Régis Behmo
d73d6732d5 feat: make it possible to override jobs in dev
Previously, job declarations were always loaded from local/docker-compose.yml
and local/docker-compose.jobs.yml. This meant that it was not possible to
override job declarations in dev mode. It is now the case, with
dev/docker-compose.jobs.yml and dev/docker-compose.jobs.override.yml. Neither
of these files exist yet... But who knows? we might need this feature one day.
In any case the code is much cleaner now.
2021-11-01 17:21:43 +01:00
Régis Behmo
02536e0f9f refactor: better runner inheritance architecture
Before, custom `docker_compose_func` arguments had to be passed to job runners.
This was not very elegant. Also, it prevented us from loading custom job files
in development.

Here, we adopt a better object-oriented approach, where context classes are
ordered hierarchically.

This paves the way for loading `dev/docker-compose.jobs.yml` files in `tutor
dev init` commands -- which will be necessary to fix permissions in dev/local
mode.
2021-11-01 17:21:43 +01:00
Régis Behmo
079fb1c9ec fix: bypass build to accelerate "local start"
Previously, we were building all images every time we ran a "local start"
command. This was causing unnecessary rebuild. Here, instead, we make use of
the `docker-compose up --build`. This means that only the required images will
be rebuilt.
2021-11-01 17:00:11 +01:00
Michael Wheeler
0a8d92f8d4 Swap incorrect documentation filenames 2021-10-28 15:57:06 +02:00
Régis Behmo
78117d16f2 chore: get rid of outdated pycryptodome ugly patch
This patch is no longer required now that the fix has been merged upstream, in
3.10.3: https://github.com/Legrandin/pycryptodome/issues/506
2021-10-25 20:19:27 +02:00
Peter Parkkali
a095a6fbc7 fix: Change memory allocation error into a warning in 'local quickstart' 2021-10-25 20:17:38 +02:00
Peter Parkkali
2549aef4dc fix: require at least 4 GB RAM on macOS for 'local quickstart' only
Limits the memory chek to the 'local quickstart' command, makes error
handling more accurate and adds warning messages for some conditions.
Also adds a mention of this in troubleshooting.rst.
2021-10-25 20:17:38 +02:00
Peter Parkkali
fb2aeefd91 fix: require at least 4 GB RAM on macOS for local commands
Adds a check in the 'local' command group that requires at least
4 GB of RAM to be allocated to Docker when running any of the
local subcommands on macOS. This addresses a common issue where
Docker's default setting (2 GB) causes startup to crash with
misleading error messages.
2021-10-25 20:17:38 +02:00
Régis Behmo
01d374d2b1 v12.1.5 (2021-10-25)
- 💥[Improvement] Change the `settheme` command such that, by default, a custom theme is assigned to the LMS and the CMS, both in production and development mode.
2021-10-25 17:54:40 +02:00
Régis Behmo
030d56f9af docs: nightly development workflow 2021-10-14 12:59:57 +02:00