6
0
mirror of https://github.com/ChristianLight/tutor.git synced 2024-12-14 06:58:21 +00:00
Commit Graph

34 Commits

Author SHA1 Message Date
Carlos Muniz
72c417da62 docs: add entry to CHANGELOG-nightly.md 2022-09-06 16:06:26 +02:00
David Ormsbee
6ac1c0d732 fix: replica set error from pymongo 3.10 -> 3.12 upgrade
The pymongo dependency for edx-platform was updated (3.10.1 to 3.12.3)
in https://github.com/openedx/edx-platform/pull/30569

This caused the following error when running the edx-platform database
migration split_modulestore_django.0002_data_migration as part of
`tutor dev quickstart`:

  pymongo.errors.ServerSelectionTimeoutError: client is configured to
  connect to a replica set named '' but this node belongs to a set named
  'None', Timeout: 30s, Topology Description: <TopologyDescription id:
  62bdbaf182687350acf1aeec, topology_type: Single, servers:
  [<ServerDescription ('mongodb', 27017) server_type: Unknown, rtt:
   None, error=ConfigurationError("client is configured to connect to a
   replica set named '' but this node belongs to a set named 'None'")>]>

This commit explicitly sets replicaSet to None to indicate that it's a
standalone MongoDB instance. I also had to remove the CONTENTSTORE entry
from auth.yml because edx-platform's devstack.py assumes it has a
non-null value (set in common.py), and devstack.py executes before
tutor's development.py can set this replicaSet value.
2022-07-05 16:44:12 +02:00
Régis Behmo
87132d27e6 Merge branch 'master' into nightly 2022-05-06 16:14:26 +02:00
Régis Behmo
85a44213b7 docs: improve changelog to highlight 3rd party contributions
With this change, we want to better highlight the contributions of
developers to Tutor. We want to publicly acknowledge the positive impact
that individuals and companies have on the development of the platform.
to that end, each changelog entry can now be suffixed with the name of
the author (individual or company) who authored the change. These names
will find their way to the release notes for every release. Eventually,
we also want to spread these release notes more widely. For instance, we
could post new releases to the forum to notify the community of
important changes.

If you have contributed to Tutor in the past, feel free to open a PR and
append your name to the changes that you made. We will not be able to
update the release notes for every release out there, but your
contributions will be acknowledged from the changelog.
2022-05-06 13:58:59 +01:00
Kyle McCormick
698f49854d build: NIGHTLY ONLY: install nightly branches of official plugins
For Tutor Nightly (and only Nightly), official plugins are now installed
from their nightly branches on GitHub instead of a version range on
PyPI. This will allow Nightly users to install all official plugins by
running:

    pip install -e ".[full]"

Notes:
  * We use the syntax `EGG @ git+REPO@nightly` because the
    more common syntax of `git+REPO@nightly#egg=EGG` does not work
    when supplied to setup.py's extras_require.
  * Unlike other plugins, tutor-license is still installed from PyPI,
    but without any version constraint. This is because tutor-license
    is a simple, closed-source plugin which activates Wizard edition
    for subscribers. It should be available in Nightly but doesn't
    need to be installed from its own bleeding-edge branch.
  * Unlike most nightly commits, this commit should NOT ever be
    reflected on master. When it comes time to merge nightly into
    master during the release of Nutmeg, this commit will need to
    be manually reverted from master.
  * Documentation updates have been made separately so that they
    can be merged into master.
2022-04-11 18:49:45 +02:00
Régis Behmo
18d3b57748 fix: remove references to edX from bulk emails
Bulk course emails used to contain many references to edX. This is
resolved here by switching to edx-ace for sending emails.

See:
https://github.com/openedx/build-test-release-wg/issues/100
https://edx.readthedocs.io/projects/edx-platform-technical/en/latest/featuretoggles.html#featuretoggle-BULK_EMAIL_SEND_USING_EDX_ACE
https://github.com/openedx/edx-platform/pull/29900
2022-03-31 15:26:15 +02:00
Régis Behmo
cbe753bc73 docs: trim changelog-nightly
Older changes have been moved to CHANGELOG.md, so we can remove them
from this file.
2022-03-31 15:26:15 +02:00
Kyle McCormick
76ef7de74f fix: lms-worker/cms-worker command update for Celery 5
Before edx-platform was upgraded to Celery 5, lms-worker and
cms-worker could be invoked using this syntax:

  celery worker --app=APP <args> --maxtasksperchild=N <args>

Since the recent Celery 5 upgrade (edx-platform commit 0588c92),
though, this fails with the messages:

  You are using `--app` as an option of the worker sub-command:
  celery worker --app celeryapp <...>
  The support for this usage was removed in Celery 5.0.
  Instead you should use `--app` as a global option:
  celery --app celeryapp worker <...>

and:

  Error: No such option: --maxtasksperchild
  (Possible options: --max-memory-per-child, --max-tasks-per-child)

So, this commit changes the lms-worker and cms-worker invocations to:

  celery --app=APP <args> --max-tasks-per-child=N <args>
2022-03-24 08:20:24 +01:00
Kyle McCormick
71b4c14d69 chore: set CMS_CFG instead of STUDIO_CFG
In the LMS/CMS Dockerfile, the env var STUDIO_CFG is set
in order to point CMS at its configuration json/yaml file.

Since https://github.com/edx/edx-platform/pull/29534
(which introduced 0013-cms-vs-studio.rst), the STUDIO_CFG
variable has been deprecated in favor of CMS_CFG.
This change updates the Dockerfile to reflect the new
preferred environment variable.

The only noticeable impact of this change is that it
will remove a depreation warning from Django startup
for tutor uses running off of Open edX master.
2022-01-20 11:22:55 +01:00
Régis Behmo
7c157eccd5 feat: upgrade to Maple
- A shared cookie domain between lms and cms is no longer recommended:
https://github.com/edx/edx-platform/blob/master/docs/guides/studio_oauth.rst
- refactor: clean mounted data folder in lms/cms. In Lilac, the
bind-mounted lms/data and cms/data folders are a mess because new
folders are created there for every new course organisation.  These
folders are empty. As far as we know they are useless... With this
change we move these folders to a dedicated "modulestore" subdirectory;
which corresponds better to the initial intent of the fs_root setting.
- fix: frontend failure during login to the lms. See:
https://github.com/openedx/build-test-release-wg/issues/104
- feat: move all forum-related code to a dedicated plugin. Forum is an
optional feature, and as such it deserves its own plugin. Starting from
Maple, users will be able to install the forum from
https://github.com/overhangio/tutor-forum/
- migrate from DCS_* session cookie settings to SESSION_*. That's
because edx-platform no longer depends on django-cookies-samesite. Close
https://github.com/openedx/build-test-release-wg/issues/110
- get rid of tons of deprecation warnings in the lms/cms
- feat: make it possible to point to themed assets. Cherry-picking this
change makes it possible to point to themed assets with a theme-agnostic
url, notably from MFEs.
- Install all official plugins as part of the `tutor[full]` package.
- Don't print error messages about loading plugins during autocompletion.
- Prompt for image building when upgrading from one release to the next.
- Add `tutor local start --skip-build` option to skip building Docker images.

Close #450.
Close #545.
2021-12-20 21:21:36 +01:00
Régis Behmo
d8d0560b9e feat: better support of caddy as a k8s load balancer
This introduces quite a few changes to make it easier to run Caddy as a load
balancer in Kubernetes:

- Make it possible to start/stop a selection of resources with ``tutor k8s
  start/stop [names...]``.
- Make it easy to deploy an independent LoadBalancer by converting the caddy
  service to a NodePort when ``ENABLE_WEB_PROXY=false``.
- Add a ``app.kubernetes.io/component: loadbalancer`` label to the LoadBalancer
  service.
- Add ``app.kubernetes.io/name`` labels to all services.
- Preserve the LoadBalancer service in ``tutor k8s stop`` commands.
- Wait for the caddy deployment to be ready before running initialisation jobs.

Close #532.
2021-12-20 21:19:10 +01:00
Régis Behmo
7c1e85ef4b security: convert NodePort to ClusterIP for better isolation
On some providers (notably: DigitalOcean) NodePort services are not exposed to
the outside world. But this is not what the Kubernetes spec describes:
https://kubernetes.io/docs/concepts/services-networking/service/#publishing-services-service-types

Thus, there is a risk that NodePort services are exposed to the outside world
in some context. To avoid this, we convert all NodePort to ClusterIP resources.
2021-12-20 21:19:10 +01:00
Régis Behmo
28edf106c4 chore: drop python 3.5 compatibility
Python 3.5 has reached end of life in September 3.5. Anyway, Tutor was not
compatible because some dev dependencies, such as astroid 2.8.3, are no longer
available in 3.5.

This means that we can now start using many python 3.6 niceties, such as
f-strings \o/
2021-12-20 21:19:10 +01:00
Régis Behmo
3d92087f3a fix: docker-compose project name in dev on nightly
Project name was incorrectly set to "tutor_dev" instead of "tutor_nightly_dev".
2021-12-20 21:19:10 +01:00
Florian Haas
9446095665 fix: Stop creating immutable resource label that breaks "tutor k8s" on Tutor version changes
Through the commonLabels directive in kustomization.yml, all resources
get a label named "app.kubernetes.io/version", which is being set to
the Tutor version at the time of initial deployment.

When the user then subsequently progresses to a new Tutor version,
Kubernetes attempts to update this label — but for Deployment,
ReplicaSet, and DaemonSet resources, this is no longer allowed as of
https://github.com/kubernetes/kubernetes/issues/50808. This causes
"tutor k8s start" (at the "kubectl apply --kustomize" step) to break
with errors such as:

Deployment.apps "redis" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{"app.kubernetes.io/instance":"openedx-JIONBLbtByCGUYgHgr4tDWu1", "app.kubernetes.io/managed-by":"tutor", "app.kubernetes.io/name":"redis", "app.kubernetes.io/part-of":"openedx", "app.kubernetes.io/version":"12.1.7"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable

Simply removing the app.kubernetes.io/version label from
kustomization.yml will permanently fix this issue for newly created
Kubernetes deployments, which will "survive" any future Tutor version
changes thereafter.

However, *existing* production Open edX deployments will need to throw
the affected Deployments away, and re-create them.

Also, add the Tutor version as a resource annotation instead, using
the commonAnnotations directive.

See also:
https://github.com/kubernetes/client-go/issues/508
https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/commonlabels/
https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/commonannotations/

Fixes #531.
2021-12-20 21:19:10 +01:00
Régis Behmo
7b5ec22e0c fix: enable plugins to implement the "caddyfile" patch
When nginx was removed in favour of caddy, we decided that plugin
implementations of the "caddyfile" patch should make use of the "port" local
variable. However, local variables are not available from inside plugin
patches, which are rendered outside of the context of the parent templates.

For a more extensive description of the problem, see:
https://github.com/overhangio/tutor-mfe/pull/23#issuecomment-964016190

We still want to make it easy for developers to decide what should the port be
for caddy hosts. To do so, we make use of environment variables that are passed
at runtime to the caddy container.

Thus, a regular plugin patch should look like this:

    {{ PLUGIN_HOST }}{$default_site_port} {
        import proxy "myplugin:8000"
    }
2021-12-20 21:19:10 +01:00
Régis Behmo
b8ab829c11 feat: move all forum-related code to a dedicated plugin
Forum is an optional feature, and as such it deserves its own plugin. Starting
from Maple, users will be able to install the forum from
https://github.com/overhangio/tutor-forum/

Close #450.
2021-12-20 21:19:10 +01:00
Régis Behmo
9964315abe chore: get rid of tutor-openedx
In the past, tutor was installed with "pip install tutor-openedx". For
some time (since v12.0.2), "tutor" was installed as a dependency of
"tutor-openedx". Now is the time to get rid of that old package.
The standard way of installing tutor is now with "pip install tutor".
2021-12-20 21:19:10 +01:00
Régis Behmo
0658be86b8 fix: always run Caddy on Kubernetes
Caddy should always be running, even when ENABLE_WEB_PROXY is false.
It's the service that should not always be running.
2021-12-20 21:19:10 +01:00
Régis Behmo
01b58d9d75 feat: run all services as unprivileged containers
With this change, containers are no longer run as "root" but as unprivileged
users. This is necessary in some environments, notably some Kubernetes
clusters.

To make this possible, we need to manually fix bind-mounted volumes in
docker-compose. This is pretty much equivalent to the behaviour in Kubernetes,
where permissions are fixed at runtime if the volume owner is incorrect. Thus,
we have a consistent behaviour between docker-compose and Kubernetes.

We achieve this by bind-mounting some repos inside "*-permissions" services.
These services run as root user on docker-compose and will fix the required
permissions, as per build/permissions/setowner.sh These services simply do not
run on Kubernetes, where we don't rely on bind-mounted volumes. There, we make
use of Kubernete's built-in volume ownership feature.

With this change, we get rid of the "openedx-dev" Docker image, in the sense
that it no longer has its own Dockerfile. Instead, the dev image is now simply
a different target in the multi-layer openedx Docker image. This makes it much
faster to build the openedx-dev image.

Because we declare the APP_USER_ID in the dev/docker-compose.yml file, we need
to pass the user ID from the host there. The only way to achieve that is with a
tutor config variable. The downside of this approach is that the
dev/docker-compose.yml file is no longer portable from one machine to the next.
We consider that this is not such a big issue, as it affects the development
environment only.

We take this opportunity to replace the base image of the "forum" image. There
is now no need to re-install ruby inside the image. The total image size is
only decreased by 10%, but re-building the image is faster.

In order to run the smtp service as non-root, we switch from namshi/smtp to
devture/exim-relay. This change should be backward-compatible.

Note that the nginx container remains privileged. We could switch to
nginxinc/nginx-unprivileged, but it's probably not worth the effort, as we are
considering to get rid of the nginx container altogether.

Close #323.
2021-12-20 21:19:10 +01:00
Régis Behmo
231bbbfe99 feat: get rid of the nginx container and services
Nginx and Caddy performed duplicate tasks. It was decided to get rid of
the nginx container, for simplification. This is a breaking change for
plugin developers. Also, applications that collect nginx logs will have
to be modified.

See:
- Corresponding TEP: https://discuss.overhang.io/t/tep-get-rid-of-the-nginx-container/2024
- the prior discussion: https://discuss.overhang.io/t/why-caddy-nginx/1952
2021-12-20 21:19:10 +01:00
Braden MacDonald
bf40724dd7 fix: a migration on nightly/master open edx now requires MongoDB 2021-12-16 15:51:13 +01:00
Régis Behmo
39ca60f168 feat: better support of caddy as a k8s load balancer
This introduces quite a few changes to make it easier to run Caddy as a load
balancer in Kubernetes:

- Make it possible to start/stop a selection of resources with ``tutor k8s
  start/stop [names...]``.
- Make it easy to deploy an independent LoadBalancer by converting the caddy
  service to a NodePort when ``ENABLE_WEB_PROXY=false``.
- Add a ``app.kubernetes.io/component: loadbalancer`` label to the LoadBalancer
  service.
- Add ``app.kubernetes.io/name`` labels to all services.
- Preserve the LoadBalancer service in ``tutor k8s stop`` commands.
- Wait for the caddy deployment to be ready before running initialisation jobs.

Close #532.
2021-12-14 11:25:41 +01:00
Régis Behmo
e5b63604de security: convert NodePort to ClusterIP for better isolation
On some providers (notably: DigitalOcean) NodePort services are not exposed to
the outside world. But this is not what the Kubernetes spec describes:
https://kubernetes.io/docs/concepts/services-networking/service/#publishing-services-service-types

Thus, there is a risk that NodePort services are exposed to the outside world
in some context. To avoid this, we convert all NodePort to ClusterIP resources.
2021-12-14 11:25:41 +01:00
Régis Behmo
1f8555b80e chore: drop python 3.5 compatibility
Python 3.5 has reached end of life in September 3.5. Anyway, Tutor was not
compatible because some dev dependencies, such as astroid 2.8.3, are no longer
available in 3.5.

This means that we can now start using many python 3.6 niceties, such as
f-strings \o/
2021-11-30 15:06:13 +01:00
Régis Behmo
a074ff34c7 fix: docker-compose project name in dev on nightly
Project name was incorrectly set to "tutor_dev" instead of "tutor_nightly_dev".
2021-11-29 14:12:22 +01:00
Florian Haas
55582575f0 fix: Stop creating immutable resource label that breaks "tutor k8s" on Tutor version changes
Through the commonLabels directive in kustomization.yml, all resources
get a label named "app.kubernetes.io/version", which is being set to
the Tutor version at the time of initial deployment.

When the user then subsequently progresses to a new Tutor version,
Kubernetes attempts to update this label — but for Deployment,
ReplicaSet, and DaemonSet resources, this is no longer allowed as of
https://github.com/kubernetes/kubernetes/issues/50808. This causes
"tutor k8s start" (at the "kubectl apply --kustomize" step) to break
with errors such as:

Deployment.apps "redis" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{"app.kubernetes.io/instance":"openedx-JIONBLbtByCGUYgHgr4tDWu1", "app.kubernetes.io/managed-by":"tutor", "app.kubernetes.io/name":"redis", "app.kubernetes.io/part-of":"openedx", "app.kubernetes.io/version":"12.1.7"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable

Simply removing the app.kubernetes.io/version label from
kustomization.yml will permanently fix this issue for newly created
Kubernetes deployments, which will "survive" any future Tutor version
changes thereafter.

However, *existing* production Open edX deployments will need to throw
the affected Deployments away, and re-create them.

Also, add the Tutor version as a resource annotation instead, using
the commonAnnotations directive.

See also:
https://github.com/kubernetes/client-go/issues/508
https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/commonlabels/
https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/commonannotations/

Fixes #531.
2021-11-25 09:09:26 +01:00
Régis Behmo
72baae0e27 fix: enable plugins to implement the "caddyfile" patch
When nginx was removed in favour of caddy, we decided that plugin
implementations of the "caddyfile" patch should make use of the "port" local
variable. However, local variables are not available from inside plugin
patches, which are rendered outside of the context of the parent templates.

For a more extensive description of the problem, see:
https://github.com/overhangio/tutor-mfe/pull/23#issuecomment-964016190

We still want to make it easy for developers to decide what should the port be
for caddy hosts. To do so, we make use of environment variables that are passed
at runtime to the caddy container.

Thus, a regular plugin patch should look like this:

    {{ PLUGIN_HOST }}{$default_site_port} {
        import proxy "myplugin:8000"
    }
2021-11-09 11:33:56 +01:00
Régis Behmo
7a3026efe6 feat: move all forum-related code to a dedicated plugin
Forum is an optional feature, and as such it deserves its own plugin. Starting
from Maple, users will be able to install the forum from
https://github.com/overhangio/tutor-forum/

Close #450.
2021-11-08 10:44:47 +01:00
Régis Behmo
999c23d1ff chore: get rid of tutor-openedx
In the past, tutor was installed with "pip install tutor-openedx". For
some time (since v12.0.2), "tutor" was installed as a dependency of
"tutor-openedx". Now is the time to get rid of that old package.
The standard way of installing tutor is now with "pip install tutor".
2021-11-02 18:26:07 +01:00
Régis Behmo
7a01f9d009 fix: always run Caddy on Kubernetes
Caddy should always be running, even when ENABLE_WEB_PROXY is false.
It's the service that should not always be running.
2021-11-01 17:00:59 +01:00
Régis Behmo
f9402f7879 feat: run all services as unprivileged containers
With this change, containers are no longer run as "root" but as unprivileged
users. This is necessary in some environments, notably some Kubernetes
clusters.

To make this possible, we need to manually fix bind-mounted volumes in
docker-compose. This is pretty much equivalent to the behaviour in Kubernetes,
where permissions are fixed at runtime if the volume owner is incorrect. Thus,
we have a consistent behaviour between docker-compose and Kubernetes.

We achieve this by bind-mounting some repos inside "*-permissions" services.
These services run as root user on docker-compose and will fix the required
permissions, as per build/permissions/setowner.sh These services simply do not
run on Kubernetes, where we don't rely on bind-mounted volumes. There, we make
use of Kubernete's built-in volume ownership feature.

With this change, we get rid of the "openedx-dev" Docker image, in the sense
that it no longer has its own Dockerfile. Instead, the dev image is now simply
a different target in the multi-layer openedx Docker image. This makes it much
faster to build the openedx-dev image.

Because we declare the APP_USER_ID in the dev/docker-compose.yml file, we need
to pass the user ID from the host there. The only way to achieve that is with a
tutor config variable. The downside of this approach is that the
dev/docker-compose.yml file is no longer portable from one machine to the next.
We consider that this is not such a big issue, as it affects the development
environment only.

We take this opportunity to replace the base image of the "forum" image. There
is now no need to re-install ruby inside the image. The total image size is
only decreased by 10%, but re-building the image is faster.

In order to run the smtp service as non-root, we switch from namshi/smtp to
devture/exim-relay. This change should be backward-compatible.

Note that the nginx container remains privileged. We could switch to
nginxinc/nginx-unprivileged, but it's probably not worth the effort, as we are
considering to get rid of the nginx container altogether.

Close #323.
2021-10-25 16:26:04 +02:00
Régis Behmo
e19f334ebb feat: get rid of the nginx container and services
Nginx and Caddy performed duplicate tasks. It was decided to get rid of
the nginx container, for simplification. This is a breaking change for
plugin developers. Also, applications that collect nginx logs will have
to be modified.

See:
- Corresponding TEP: https://discuss.overhang.io/t/tep-get-rid-of-the-nginx-container/2024
- the prior discussion: https://discuss.overhang.io/t/why-caddy-nginx/1952
2021-10-25 16:18:42 +02:00
Régis Behmo
030d56f9af docs: nightly development workflow 2021-10-14 12:59:57 +02:00