2019-05-09 07:51:06 +00:00
|
|
|
---
|
|
|
|
|
apiVersion: kustomize.config.k8s.io/v1beta1
|
|
|
|
|
kind: Kustomization
|
|
|
|
|
resources:
|
|
|
|
|
- k8s/namespace.yml
|
|
|
|
|
- k8s/deployments.yml
|
Improve job running in local and k8s
Running jobs was previously done with "exec". This was because it
allowed us to avoid copying too much container specification information
from the docker-compose/deployments files to the jobs files. However,
this was limiting:
- In order to run a job, the corresponding container had to be running.
This was particularly painful in Kubernetes, where containers are
crashing as long as migrations are not correctly run.
- Containers in which we need to run jobs needed to be present in the
docker-compose/deployments files. This is unnecessary, for example when
mysql is disabled, or in the case of the certbot container.
Now, we create dedicated jobs files, both for local and k8s deployment.
This introduces a little redundancy, but not too much. Note that
dependent containers are not listed in the docker-compose.jobs.yml file,
so an actual platform is still supposed to be running when we launch the
jobs.
This also introduces a subtle change: now, jobs go through the container
entrypoint prior to running. This is probably a good thing, as it will
avoid forgetting about incorrect environment variables.
In k8s, we find ourselves interacting way too much with the kubectl
utility. Parsing output from the CLI is a pain. So we need to switch to
the native kubernetes client library.
2020-03-25 17:47:36 +00:00
|
|
|
- k8s/jobs.yml
|
2019-05-09 07:51:06 +00:00
|
|
|
- k8s/services.yml
|
|
|
|
|
- k8s/volumes.yml
|
2019-07-11 02:32:40 +00:00
|
|
|
{{ patch("kustomization-resources") }}
|
2019-05-09 07:51:06 +00:00
|
|
|
|
|
|
|
|
# namespace to deploy all Resources to
|
|
|
|
|
namespace: {{ K8S_NAMESPACE }}
|
|
|
|
|
|
fix: Stop creating immutable resource label that breaks "tutor k8s" on Tutor version changes
Through the commonLabels directive in kustomization.yml, all resources
get a label named "app.kubernetes.io/version", which is being set to
the Tutor version at the time of initial deployment.
When the user then subsequently progresses to a new Tutor version,
Kubernetes attempts to update this label — but for Deployment,
ReplicaSet, and DaemonSet resources, this is no longer allowed as of
https://github.com/kubernetes/kubernetes/issues/50808. This causes
"tutor k8s start" (at the "kubectl apply --kustomize" step) to break
with errors such as:
Deployment.apps "redis" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{"app.kubernetes.io/instance":"openedx-JIONBLbtByCGUYgHgr4tDWu1", "app.kubernetes.io/managed-by":"tutor", "app.kubernetes.io/name":"redis", "app.kubernetes.io/part-of":"openedx", "app.kubernetes.io/version":"12.1.7"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable
Simply removing the app.kubernetes.io/version label from
kustomization.yml will permanently fix this issue for newly created
Kubernetes deployments, which will "survive" any future Tutor version
changes thereafter.
However, *existing* production Open edX deployments will need to throw
the affected Deployments away, and re-create them.
Also, add the Tutor version as a resource annotation instead, using
the commonAnnotations directive.
See also:
https://github.com/kubernetes/client-go/issues/508
https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/commonlabels/
https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/commonannotations/
Fixes #531.
2021-11-18 13:07:40 +00:00
|
|
|
# annotations added to all Resources
|
|
|
|
|
# https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/commonannotations/
|
|
|
|
|
commonAnnotations:
|
|
|
|
|
app.kubernetes.io/version: {{ TUTOR_VERSION }}
|
|
|
|
|
|
|
|
|
|
# labels (and label selectors) added to all Resources
|
2019-05-09 07:51:06 +00:00
|
|
|
# https://kubernetes.io/docs/concepts/overview/working-with-objects/common-labels/
|
fix: Stop creating immutable resource label that breaks "tutor k8s" on Tutor version changes
Through the commonLabels directive in kustomization.yml, all resources
get a label named "app.kubernetes.io/version", which is being set to
the Tutor version at the time of initial deployment.
When the user then subsequently progresses to a new Tutor version,
Kubernetes attempts to update this label — but for Deployment,
ReplicaSet, and DaemonSet resources, this is no longer allowed as of
https://github.com/kubernetes/kubernetes/issues/50808. This causes
"tutor k8s start" (at the "kubectl apply --kustomize" step) to break
with errors such as:
Deployment.apps "redis" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{"app.kubernetes.io/instance":"openedx-JIONBLbtByCGUYgHgr4tDWu1", "app.kubernetes.io/managed-by":"tutor", "app.kubernetes.io/name":"redis", "app.kubernetes.io/part-of":"openedx", "app.kubernetes.io/version":"12.1.7"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable
Simply removing the app.kubernetes.io/version label from
kustomization.yml will permanently fix this issue for newly created
Kubernetes deployments, which will "survive" any future Tutor version
changes thereafter.
However, *existing* production Open edX deployments will need to throw
the affected Deployments away, and re-create them.
Also, add the Tutor version as a resource annotation instead, using
the commonAnnotations directive.
See also:
https://github.com/kubernetes/client-go/issues/508
https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/commonlabels/
https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/commonannotations/
Fixes #531.
2021-11-18 13:07:40 +00:00
|
|
|
# https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/commonlabels/
|
2019-05-09 07:51:06 +00:00
|
|
|
commonLabels:
|
|
|
|
|
app.kubernetes.io/instance: openedx-{{ ID }}
|
|
|
|
|
app.kubernetes.io/part-of: openedx
|
|
|
|
|
app.kubernetes.io/managed-by: tutor
|
2019-07-11 02:32:40 +00:00
|
|
|
{{ patch("kustomization-commonlabels")|indent(2) }}
|
2019-05-09 07:51:06 +00:00
|
|
|
|
|
|
|
|
configMapGenerator:
|
2020-09-17 10:53:14 +00:00
|
|
|
- name: caddy-config
|
|
|
|
|
files:
|
|
|
|
|
- apps/caddy/Caddyfile
|
2022-01-27 09:10:53 +00:00
|
|
|
options:
|
|
|
|
|
labels:
|
|
|
|
|
app.kubernetes.io/name: caddy
|
2019-05-09 07:51:06 +00:00
|
|
|
- name: openedx-settings-lms
|
|
|
|
|
files:{% for file in "apps/openedx/settings/lms"|walk_templates %}
|
|
|
|
|
- {{ file }}{% endfor %}
|
2021-10-18 09:43:40 +00:00
|
|
|
options:
|
|
|
|
|
labels:
|
|
|
|
|
app.kubernetes.io/name: openedx
|
2019-05-09 07:51:06 +00:00
|
|
|
- name: openedx-settings-cms
|
|
|
|
|
files:{% for file in "apps/openedx/settings/cms"|walk_templates %}
|
|
|
|
|
- {{ file }}{% endfor %}
|
2021-10-18 09:43:40 +00:00
|
|
|
options:
|
|
|
|
|
labels:
|
|
|
|
|
app.kubernetes.io/name: openedx
|
2019-05-09 07:51:06 +00:00
|
|
|
- name: openedx-config
|
|
|
|
|
files:{% for file in "apps/openedx/config"|walk_templates %}
|
|
|
|
|
- {{ file }}{% endfor %}
|
2021-10-18 09:43:40 +00:00
|
|
|
options:
|
|
|
|
|
labels:
|
|
|
|
|
app.kubernetes.io/name: openedx
|
2023-02-25 03:01:35 +00:00
|
|
|
- name: openedx-uwsgi-config
|
|
|
|
|
files:
|
|
|
|
|
- apps/openedx/uwsgi.ini
|
|
|
|
|
options:
|
|
|
|
|
labels:
|
|
|
|
|
app.kubernetes.io/name: openedx
|
2020-09-17 10:53:14 +00:00
|
|
|
- name: redis-config
|
|
|
|
|
files:
|
|
|
|
|
- apps/redis/redis.conf
|
2021-10-18 09:43:40 +00:00
|
|
|
options:
|
|
|
|
|
labels:
|
|
|
|
|
app.kubernetes.io/name: redis
|
2019-07-02 20:16:44 +00:00
|
|
|
{{ patch("kustomization-configmapgenerator") }}
|
2019-07-11 02:32:40 +00:00
|
|
|
|
2022-07-13 21:26:05 +00:00
|
|
|
{%- if patch("k8s-override") or patch("kustomization-patches-strategic-merge") %}
|
2022-06-07 13:59:25 +00:00
|
|
|
patchesStrategicMerge:
|
2022-07-13 20:49:07 +00:00
|
|
|
{%- if patch("k8s-override") %}
|
2022-06-07 13:59:25 +00:00
|
|
|
- k8s/override.yml
|
2022-07-13 20:49:07 +00:00
|
|
|
{%- endif %}
|
2022-06-07 13:59:25 +00:00
|
|
|
{{ patch("kustomization-patches-strategic-merge") }}
|
2022-07-13 20:49:07 +00:00
|
|
|
{%- endif %}
|
2022-06-07 13:59:25 +00:00
|
|
|
|
2021-10-18 09:43:40 +00:00
|
|
|
{{ patch("kustomization") }}
|
