6
0
mirror of https://github.com/ChristianLight/tutor.git synced 2024-12-12 22:27:47 +00:00
Commit Graph

1180 Commits

Author SHA1 Message Date
Régis Behmo
c61accedfc refactor: move upgrade code to separate modules
This results in clearer code.
2022-01-08 19:07:26 +01:00
Régis Behmo
9fc928a711 fix: update env prior to rebuilding images during upgrade
The fact that the environment was not up-to-date was causing errors and
confusion:
https://discuss.overhang.io/t/install-maple-importerror-cannot-import-name-removedindjango40warning/2255/6
2022-01-08 19:07:26 +01:00
Régis Behmo
59291eed52 fix: ownership of mysql data
In theory, we can assign ownership of mysql data to just any user. But in
Lilac, mysql was running with user 999. When upgrading to Maple, on Kubernetes,
the fsGroupChangePolicy was causing a change of the data *group* (to 1000) but
not of the user. This was causing a crash with the following error:

    [ERROR] InnoDB: The error means mysqld does not have the access rights to the directory.
2022-01-08 19:07:26 +01:00
Régis Behmo
70dbfcb6fb fix: upgrade from Lilac on k8s
When upgrading from Lilac, all services break with the following error:

    Service "***" is invalid: spec.ports[0].nodePort: Forbidden: may not be used when `type` is 'ClusterIP'

Upgrading deployments fails as well:

    Deployment.apps "***" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{"app.kubernetes.io/instance":"openedx-********", "app.kubernetes.io/managed-by":"tutor", "app.kubernetes.io/name":"***", "app.kubernetes.io/part-of":"openedx"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable

That's because deployments and services need to be deleted as part of the Maple
upgrade. So that's what we do as part of `tutor k8s upgrade --from=lilac`. And
we take the opportunity to:

1. Run upgrade as part of quickstart, when necessary.
2. Default to lilac during `tutor k8s upgrade`.

Close #551.
2022-01-08 19:07:26 +01:00
Régis Behmo
43d5da83e4 fix: utils tests on macOS
test_utils tests were failing on macOS when the settings file was properly
defined and present.

Close #560.
2022-01-08 18:48:50 +01:00
Jayram Nai
37be2bf122 docs: fixed typo 2022-01-08 17:58:26 +01:00
Ned Batchelder
f2260006bd build: 'make help' works nicer on Mac
sed on Mac doesn't seem to understand \n.

While I was in here, I fixed some typos.
2022-01-06 10:44:53 +01:00
Ned Batchelder
ac39f9e6c3 Minor fix to install.rst 2022-01-05 10:05:12 +01:00
Régis Behmo
f4d866db81 v13.0.3 (2022-01-04)
- [Security] Upgrade Django to 3.2.11 in edx-platform.
- [Security] Prevent non-staff users from searching usernames by email by
  abusing the logout url.
2022-01-04 17:33:37 +01:00
Régis Behmo
699ef4c6d5 security: upgrade django to 3.2.11 in edx-platform
See release notes: https://docs.djangoproject.com/en/3.2/releases/3.2.11/
2022-01-04 16:38:23 +01:00
Régis Behmo
622e7cc097 security: prevent searching user emails by name (again)
This time, the security fix addresses an issue in the logout url.
2022-01-04 15:43:30 +01:00
Régis Behmo
97f7d5a1e7 fix: unit tests in nightly
Unit tests were breaking in nightly branch.
2022-01-04 14:37:37 +01:00
alex.soh
72843c06f9 refactor: add code coverage, cover CLI commands with tests 2022-01-04 13:40:33 +01:00
Braden MacDonald
dbb79c0fa0 docs: Address review comments 2022-01-03 06:56:58 +01:00
Braden MacDonald
c240aec711 docs: Document how to run Tutor on ARM-based systems 2022-01-03 06:56:58 +01:00
Derek Crosson
4488f27603 chore: update android app repository url 2022-01-03 06:46:04 +01:00
Régis Behmo
94941d0387 v13.0.2
- [Security] Prevent non-staff users from searching usernames by email.
2021-12-22 00:46:13 +01:00
Régis Behmo
553000e5f5 security: fix search username by email
Without this patch, it is possible to search for account info including
username by using the email of a learner. This fix disallows searching using
email by regular users and restricts this feature to only staff and superusers.
2021-12-22 00:45:26 +01:00
Régis Behmo
4bfaa80bd3 v13.0.1
- [Fix] Missing requirements file in `pip install tutor[full]`.
2021-12-20 23:00:12 +01:00
Régis Behmo
14c40376ec fix: missing file in pip install tutor[full]
The missing requirement file is causing the install from pypi to fail.
2021-12-20 22:59:21 +01:00
Régis Behmo
7c157eccd5 feat: upgrade to Maple
- A shared cookie domain between lms and cms is no longer recommended:
https://github.com/edx/edx-platform/blob/master/docs/guides/studio_oauth.rst
- refactor: clean mounted data folder in lms/cms. In Lilac, the
bind-mounted lms/data and cms/data folders are a mess because new
folders are created there for every new course organisation.  These
folders are empty. As far as we know they are useless... With this
change we move these folders to a dedicated "modulestore" subdirectory;
which corresponds better to the initial intent of the fs_root setting.
- fix: frontend failure during login to the lms. See:
https://github.com/openedx/build-test-release-wg/issues/104
- feat: move all forum-related code to a dedicated plugin. Forum is an
optional feature, and as such it deserves its own plugin. Starting from
Maple, users will be able to install the forum from
https://github.com/overhangio/tutor-forum/
- migrate from DCS_* session cookie settings to SESSION_*. That's
because edx-platform no longer depends on django-cookies-samesite. Close
https://github.com/openedx/build-test-release-wg/issues/110
- get rid of tons of deprecation warnings in the lms/cms
- feat: make it possible to point to themed assets. Cherry-picking this
change makes it possible to point to themed assets with a theme-agnostic
url, notably from MFEs.
- Install all official plugins as part of the `tutor[full]` package.
- Don't print error messages about loading plugins during autocompletion.
- Prompt for image building when upgrading from one release to the next.
- Add `tutor local start --skip-build` option to skip building Docker images.

Close #450.
Close #545.
2021-12-20 21:21:36 +01:00
Régis Behmo
62ddc01cdc feat: k8s scale command + tutorial
- Add a `tutor k8s scale lms 11` command
- Create a "Running Open edX at scale" tutorial
2021-12-20 21:19:10 +01:00
Régis Behmo
d8d0560b9e feat: better support of caddy as a k8s load balancer
This introduces quite a few changes to make it easier to run Caddy as a load
balancer in Kubernetes:

- Make it possible to start/stop a selection of resources with ``tutor k8s
  start/stop [names...]``.
- Make it easy to deploy an independent LoadBalancer by converting the caddy
  service to a NodePort when ``ENABLE_WEB_PROXY=false``.
- Add a ``app.kubernetes.io/component: loadbalancer`` label to the LoadBalancer
  service.
- Add ``app.kubernetes.io/name`` labels to all services.
- Preserve the LoadBalancer service in ``tutor k8s stop`` commands.
- Wait for the caddy deployment to be ready before running initialisation jobs.

Close #532.
2021-12-20 21:19:10 +01:00
Régis Behmo
7c1e85ef4b security: convert NodePort to ClusterIP for better isolation
On some providers (notably: DigitalOcean) NodePort services are not exposed to
the outside world. But this is not what the Kubernetes spec describes:
https://kubernetes.io/docs/concepts/services-networking/service/#publishing-services-service-types

Thus, there is a risk that NodePort services are exposed to the outside world
in some context. To avoid this, we convert all NodePort to ClusterIP resources.
2021-12-20 21:19:10 +01:00
Régis Behmo
db818ab4a1 fix: do not save the full config on "plugins en/disable"
This was saving the full configuration to config.yml, resulting in many
incorrect configuration values...
2021-12-20 21:19:10 +01:00
Régis Behmo
d82546adde fix: breaking "config save"
"config save" was not loading the full configuration prior to saving the
environment.
2021-12-20 21:19:10 +01:00
Régis Behmo
c40e682f5d refactor: clarify configuration management
Previously, configuration management was very confusing because we kept mixing
"base" and "defaults" configuration:

- It was difficult to make the difference between core settings that were
  necessary (e.g: passwords) as opposed to others that could simply be
  defaulted to.
- The order of settings in config.yml mattered: config entries that depended on
  other needed to be defined later. As a consequence, Tutor was not compatible
  with Python 3.5, where dict entries are not sorted.
2021-12-20 21:19:10 +01:00
Régis Behmo
b18c9dc4f8 fix: lint unused arguments in code base 2021-12-20 21:19:10 +01:00
Régis Behmo
28edf106c4 chore: drop python 3.5 compatibility
Python 3.5 has reached end of life in September 3.5. Anyway, Tutor was not
compatible because some dev dependencies, such as astroid 2.8.3, are no longer
available in 3.5.

This means that we can now start using many python 3.6 niceties, such as
f-strings \o/
2021-12-20 21:19:10 +01:00
Régis Behmo
3d92087f3a fix: docker-compose project name in dev on nightly
Project name was incorrectly set to "tutor_dev" instead of "tutor_nightly_dev".
2021-12-20 21:19:10 +01:00
Florian Haas
9446095665 fix: Stop creating immutable resource label that breaks "tutor k8s" on Tutor version changes
Through the commonLabels directive in kustomization.yml, all resources
get a label named "app.kubernetes.io/version", which is being set to
the Tutor version at the time of initial deployment.

When the user then subsequently progresses to a new Tutor version,
Kubernetes attempts to update this label — but for Deployment,
ReplicaSet, and DaemonSet resources, this is no longer allowed as of
https://github.com/kubernetes/kubernetes/issues/50808. This causes
"tutor k8s start" (at the "kubectl apply --kustomize" step) to break
with errors such as:

Deployment.apps "redis" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{"app.kubernetes.io/instance":"openedx-JIONBLbtByCGUYgHgr4tDWu1", "app.kubernetes.io/managed-by":"tutor", "app.kubernetes.io/name":"redis", "app.kubernetes.io/part-of":"openedx", "app.kubernetes.io/version":"12.1.7"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable

Simply removing the app.kubernetes.io/version label from
kustomization.yml will permanently fix this issue for newly created
Kubernetes deployments, which will "survive" any future Tutor version
changes thereafter.

However, *existing* production Open edX deployments will need to throw
the affected Deployments away, and re-create them.

Also, add the Tutor version as a resource annotation instead, using
the commonAnnotations directive.

See also:
https://github.com/kubernetes/client-go/issues/508
https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/commonlabels/
https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/commonannotations/

Fixes #531.
2021-12-20 21:19:10 +01:00
Régis Behmo
0917da59bf fix: don't attempt to security patch edx-platform 2021-12-20 21:19:10 +01:00
Régis Behmo
dc5307ad75 fix: get rid of useless redis folder in permission setting
Adding these volumes was a mistake.
2021-12-20 21:19:10 +01:00
Régis Behmo
c117a300aa fix: disable forum feature by default 2021-12-20 21:19:10 +01:00
Régis Behmo
0c1d4ebf51 fix: https test 2021-12-20 21:19:10 +01:00
Régis Behmo
7b5ec22e0c fix: enable plugins to implement the "caddyfile" patch
When nginx was removed in favour of caddy, we decided that plugin
implementations of the "caddyfile" patch should make use of the "port" local
variable. However, local variables are not available from inside plugin
patches, which are rendered outside of the context of the parent templates.

For a more extensive description of the problem, see:
https://github.com/overhangio/tutor-mfe/pull/23#issuecomment-964016190

We still want to make it easy for developers to decide what should the port be
for caddy hosts. To do so, we make use of environment variables that are passed
at runtime to the caddy container.

Thus, a regular plugin patch should look like this:

    {{ PLUGIN_HOST }}{$default_site_port} {
        import proxy "myplugin:8000"
    }
2021-12-20 21:19:10 +01:00
Régis Behmo
b8ab829c11 feat: move all forum-related code to a dedicated plugin
Forum is an optional feature, and as such it deserves its own plugin. Starting
from Maple, users will be able to install the forum from
https://github.com/overhangio/tutor-forum/

Close #450.
2021-12-20 21:19:10 +01:00
Régis Behmo
6f7457f6b3 doc: remove now irrelevant comment 2021-12-20 21:19:10 +01:00
Régis Behmo
9964315abe chore: get rid of tutor-openedx
In the past, tutor was installed with "pip install tutor-openedx". For
some time (since v12.0.2), "tutor" was installed as a dependency of
"tutor-openedx". Now is the time to get rid of that old package.
The standard way of installing tutor is now with "pip install tutor".
2021-12-20 21:19:10 +01:00
Régis Behmo
0658be86b8 fix: always run Caddy on Kubernetes
Caddy should always be running, even when ENABLE_WEB_PROXY is false.
It's the service that should not always be running.
2021-12-20 21:19:10 +01:00
Régis Behmo
ce5310db06 fix: permissions image name 2021-12-20 21:19:10 +01:00
Régis Behmo
01b58d9d75 feat: run all services as unprivileged containers
With this change, containers are no longer run as "root" but as unprivileged
users. This is necessary in some environments, notably some Kubernetes
clusters.

To make this possible, we need to manually fix bind-mounted volumes in
docker-compose. This is pretty much equivalent to the behaviour in Kubernetes,
where permissions are fixed at runtime if the volume owner is incorrect. Thus,
we have a consistent behaviour between docker-compose and Kubernetes.

We achieve this by bind-mounting some repos inside "*-permissions" services.
These services run as root user on docker-compose and will fix the required
permissions, as per build/permissions/setowner.sh These services simply do not
run on Kubernetes, where we don't rely on bind-mounted volumes. There, we make
use of Kubernete's built-in volume ownership feature.

With this change, we get rid of the "openedx-dev" Docker image, in the sense
that it no longer has its own Dockerfile. Instead, the dev image is now simply
a different target in the multi-layer openedx Docker image. This makes it much
faster to build the openedx-dev image.

Because we declare the APP_USER_ID in the dev/docker-compose.yml file, we need
to pass the user ID from the host there. The only way to achieve that is with a
tutor config variable. The downside of this approach is that the
dev/docker-compose.yml file is no longer portable from one machine to the next.
We consider that this is not such a big issue, as it affects the development
environment only.

We take this opportunity to replace the base image of the "forum" image. There
is now no need to re-install ruby inside the image. The total image size is
only decreased by 10%, but re-building the image is faster.

In order to run the smtp service as non-root, we switch from namshi/smtp to
devture/exim-relay. This change should be backward-compatible.

Note that the nginx container remains privileged. We could switch to
nginxinc/nginx-unprivileged, but it's probably not worth the effort, as we are
considering to get rid of the nginx container altogether.

Close #323.
2021-12-20 21:19:10 +01:00
Régis Behmo
231bbbfe99 feat: get rid of the nginx container and services
Nginx and Caddy performed duplicate tasks. It was decided to get rid of
the nginx container, for simplification. This is a breaking change for
plugin developers. Also, applications that collect nginx logs will have
to be modified.

See:
- Corresponding TEP: https://discuss.overhang.io/t/tep-get-rid-of-the-nginx-container/2024
- the prior discussion: https://discuss.overhang.io/t/why-caddy-nginx/1952
2021-12-20 21:19:10 +01:00
Régis Behmo
670552ca5f fix: lms 500 error caused by missing LANGUAGE_COOKIE_NAME setting
See also: https://github.com/overhangio/tutor/pull/507
Upstream fix: https://github.com/edx/edx-platform/pull/29096
2021-12-20 21:19:10 +01:00
Régis Behmo
01fb62bf29 fix: running mongodb locally and on k8s 2021-12-20 21:19:10 +01:00
Régis Behmo
d960e18f28 chore: upgrade elasticsearch/mongodb/redis
Open edX master now runs elasticsearch 7.10 and mongodb 4.2. Redis also
received a minor upgrade.
2021-12-20 21:19:10 +01:00
Braden MacDonald
1f46ed39c5 fix: openedx image wasn't building on ARM64 due to missing libgeos-dev 2021-12-14 12:10:49 +01:00
Régis Behmo
e89ff49223 v12.2.0
- 💥[Improvement] Fail on incorrect image name argument in `images
build/pull/push/printtag` commands.
- [Bugfix] Remove trailing slashes in docker-compose files for
[compatibility with docker-compose v2 in
WSL](https://github.com/docker/compose/issues/8558).
- [Improvement] `settheme` now works with preview domain.
- [Feature] Allow specifying extra pip packages through config.yml.
2021-12-08 11:29:52 +01:00
Régis Behmo
79c6d122e2 fix: incorrect "from" address in course bulk emails
Upstream PR: https://github.com/edx/edx-platform/pull/29001
See discussion:
https://discuss.overhang.io/t/sending-bulk-email-triggers-smtprecipientsrefused-error/1923
Related issue:
https://github.com/openedx/build-test-release-wg/issues/102
2021-12-08 11:29:52 +01:00
Crist Ye
ddcfab2ddb docs: improve plugin defaults readability 2021-12-08 11:06:03 +01:00