fix: Stop creating immutable resource label that breaks "tutor k8s" on Tutor version changes

Through the commonLabels directive in kustomization.yml, all resources
get a label named "app.kubernetes.io/version", which is being set to
the Tutor version at the time of initial deployment.

When the user then subsequently progresses to a new Tutor version,
Kubernetes attempts to update this label — but for Deployment,
ReplicaSet, and DaemonSet resources, this is no longer allowed as of
https://github.com/kubernetes/kubernetes/issues/50808. This causes
"tutor k8s start" (at the "kubectl apply --kustomize" step) to break
with errors such as:

Deployment.apps "redis" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{"app.kubernetes.io/instance":"openedx-JIONBLbtByCGUYgHgr4tDWu1", "app.kubernetes.io/managed-by":"tutor", "app.kubernetes.io/name":"redis", "app.kubernetes.io/part-of":"openedx", "app.kubernetes.io/version":"12.1.7"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable

Simply removing the app.kubernetes.io/version label from
kustomization.yml will permanently fix this issue for newly created
Kubernetes deployments, which will "survive" any future Tutor version
changes thereafter.

However, *existing* production Open edX deployments will need to throw
the affected Deployments away, and re-create them.

Also, add the Tutor version as a resource annotation instead, using
the commonAnnotations directive.

See also:
https://github.com/kubernetes/client-go/issues/508
https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/commonlabels/
https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/commonannotations/

Fixes #531.

CHANGELOG-nightly.md

@@ -2,6 +2,7 @@
 
 Note: Breaking changes between versions are indicated by "💥".
 
+- 💥[Bugfix] No longer track the Tutor version number in resource labels (and label selectors, which breaks the update of Deployment resources), but instead do so in resource annotations.
 - [Bugfix] Make it possible for plugins to implement the "caddyfile" patch without relying on the "port" local variable.
 - 💥[Improvement] Move the Open edX forum to a [dedicated plugin](https://github.com/overhangio/tutor-forum/) (#450).
 - 💥[Improvement] Get rid of the "tutor-openedx" package, which is no longer supported.

tutor/templates/kustomization.yml

@@ -12,11 +12,16 @@ resources:
 # namespace to deploy all Resources to
 namespace: {{ K8S_NAMESPACE }}
 
-# labels added to all Resources
+# annotations added to all Resources
+# https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/commonannotations/
+commonAnnotations:
+  app.kubernetes.io/version: {{ TUTOR_VERSION }}
+
+# labels (and label selectors) added to all Resources
 # https://kubernetes.io/docs/concepts/overview/working-with-objects/common-labels/
+# https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/commonlabels/
 commonLabels:
   app.kubernetes.io/instance: openedx-{{ ID }}
-  app.kubernetes.io/version: {{ TUTOR_VERSION }}
   app.kubernetes.io/part-of: openedx
   app.kubernetes.io/managed-by: tutor
   {{ patch("kustomization-commonlabels")|indent(2) }}