christianlight/tutor
6
0
Fork 0
mirror of https://github.com/ChristianLight/tutor.git synced 2024-12-13 14:43:03 +00:00
Code Issues Releases Activity

Merge remote-tracking branch 'origin/master' into nightly

Browse Source
This commit is contained in:
Overhang.IO 2022-01-08 18:21:08 +00:00
commit 61bde7300f
19 changed files with 585 additions and 359 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

11
CHANGELOG.md
View File

@ -4,6 +4,15 @@ Note: Breaking changes between versions are indicated by "💥".
## Unreleased
## v13.1.0 (2022-01-08)
- [Improvement] Provide much more comprehensive instructions when upgrading.
- [Bugfix] During upgrade, make sure that environment is up-to-date prior to prompting to rebuild the custom images.
- [Bugfix] Fix ownership of mysql data, in particular when upgrading a Kubernetes cluster to Maple.
- [Bugfix] Ensure that ``tutor k8s upgrade`` is run during ``tutor k8s quickstart``, when necessary.
- 💥[Bugfix] By default, detect the current version during ``tutor k8s/local upgrade``.
- [Bugfix] Fix upgrading from Lilac to Maple on Kubernetes by deleting deployments and services.
## v13.0.3 (2022-01-04)
- [Security] Upgrade Django to 3.2.11 in edx-platform.
@ -15,7 +24,7 @@ Note: Breaking changes between versions are indicated by "💥".
## v13.0.1 (2021-12-20)
- [Fix] Missing requirements file in `pip install tutor[full]`.
- [Bugfix] Missing requirements file in `pip install tutor[full]`.
## v13.0.0 (2021-12-20)

6
docs/_release_description.md
View File

@ -1,4 +1,8 @@
Tutor can be installed simply by downloading the compiled binaries:
Install this version from pip with::
pip install tutor[full]==TUTOR_VERSION
Or download the compiled binaries::
sudo curl -L "https://github.com/overhangio/tutor/releases/download/TUTOR_VERSION/tutor-$(uname -s)_$(uname -m)" -o /usr/local/bin/tutor
sudo chmod 0755 /usr/local/bin/tutor

2
docs/configuration.rst
View File

@ -79,7 +79,7 @@ You may want to pull/push images from/to a custom docker registry. For instance,
Open edX customisation
~~~~~~~~~~~~~~~~~~~~~~
- ``OPENEDX_COMMON_VERSION`` (default: ``"open-release/maple.beta2"``)
- ``OPENEDX_COMMON_VERSION`` (default: ``"open-release/maple.1"``)
This defines the default version that will be pulled from all Open edX git repositories.

2
docs/dev.rst
View File

@ -137,7 +137,7 @@ Following the instructions :ref:`above <bind_mounts>` on how to bind-mount direc
If you choose any but the first solution above, you will have to make sure that your fork works with Tutor.
First of all, you should make sure that you are working off the ``open-release/maple.beta2`` tag. See the :ref:`fork edx-platform section <edx_platform_fork>` for more information.
First of all, you should make sure that you are working off the ``open-release/maple.1`` tag. See the :ref:`fork edx-platform section <edx_platform_fork>` for more information.
Then, you should run the following commands::

1
docs/index.rst
View File

@ -37,6 +37,7 @@
Source code <https://github.com/overhangio/tutor>
Community forums <https://discuss.overhang.io>
Pypi releases <https://pypi.org/project/tutor>
Changelog <https://github.com/overhangio/tutor/blob/master/CHANGELOG.md>
Source code
-----------

38
docs/install.rst
View File

@ -87,11 +87,45 @@ Tutor can be launched on Amazon Web Services very quickly with the `official Tut
Upgrading
---------
With Tutor, it is very easy to upgrade to a more recent Open edX or Tutor release. Just install the latest ``tutor`` version (using either methods above) and run the ``quickstart`` command again. If you have :ref:`customised <configuration_customisation>` your docker images, you will have to re-build them prior to running ``quickstart``.
To upgrade Open edX or benefit from the latest features and bug fixes, you should simply upgrade Tutor. Start by upgrading the "tutor" package and its dependencies::
``quickstart`` should take care of automatically running the upgrade process. If for some reason you need to *manually* upgrade from an Open edX release to the next, you should run ``tutor local upgrade``. For instance, to upgrade from Lilac to Maple, run::
pip install --upgrade tutor[full]
Then run the ``quickstart`` command again. Depending on your deployment target, run either::
tutor local quickstart # for local installations
tutor k8s quickstart # for Kubernetes installation
Upgrading with custom Docker images
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
If you run :ref:`customised <configuration_customisation>` Docker images, you need to rebuild them prior to running ``quickstart``::
tutor config save
tutor images build all # specify here the images that you need to build
tutor local quickstart
Upgrading to a new Open edX release
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Major Open edX releases are published twice a year, in June and December, by the Open edX `Build/Test/Release working group <https://discuss.openedx.org/c/working-groups/build-test-release/30>`__. When a new Open edX release comes out, Tutor gets a major version bump (see :ref:`versioning`). Such an upgrade typically includes multiple breaking changes. Any upgrade is final, because downgrading is not supported. Thus, when upgrading your platform from one major version to the next, it is strongly recommended to do the following:
1. Read the changes listed in the `CHANGELOG.md <https://github.com/overhangio/tutor/blob/master/CHANGELOG.md>`__ file. Breaking changes are identified by a "💥".
2. Perform a backup. On a local installation, this is typically done with::
tutor local stop
sudo rsync -avr "$(tutor config printroot)"/ /tmp/tutor-backup/
3. If you created custom plugins, make sure that they are compatible with the newer release.
4. Test the new release in a sandboxed environment.
5. If you are running edx-platform, or some other repository from a custom branch, then you should rebase (and test) your changes on top of the latest release tag (see :ref:`edx_platform_fork`).
The process for upgrading from one major release to the next works similarly to any other upgrade, with the ``quickstart`` command (see above). The single difference is that if the ``quickstart`` command detects that your tutor environment was generated with an older release, it will perform a few release-specific upgrade steps. These extra upgrade steps will be performed just once. But they will be ignored if you updated your local environment (for instance: with ``tutor config save``) prior to running ``quickstart``. This situation typically occurs if you need to re-build some Docker images (see above). In such a case, you should make use of the ``upgrade`` command. For instance, to upgrade a local installation from Lilac to Maple and rebuild some Docker images, run::
tutor config save
tutor images build all # list the images that should be rebuilt here
tutor local upgrade --from=lilac
tutor local quickstart
.. _autocomplete:

10
docs/quickstart.rst
View File

@ -3,14 +3,14 @@
Quickstart (1-click install)
----------------------------
1. `Download <https://github.com/overhangio/tutor/releases>`_ the latest stable release of Tutor and place the ``tutor`` executable in your path. From the command line:
.. include:: download/binary.rst
Or:
1. Install the latest stable release of Tutor from pip:
.. include:: download/pip.rst
Or `download <https://github.com/overhangio/tutor/releases>`_ the pre-compiled binary and place the ``tutor`` executable in your path:
.. include:: download/binary.rst
2. Run ``tutor local quickstart``
3. You're done!

4
setup.py
View File

@ -1,8 +1,9 @@
import io
import os
from setuptools import find_packages, setup
from typing import Dict, List
from setuptools import find_packages, setup
HERE = os.path.abspath(os.path.dirname(__file__))
@ -30,6 +31,7 @@ def load_requirements(filename: str) -> List[str]:
) as f:
return [line.strip() for line in f if is_requirement(line)]
def is_requirement(line: str) -> bool:
return not (line.strip() == "" or line.startswith("#"))

72
tests/test_env.py
View File

@ -3,9 +3,11 @@ import tempfile
import unittest
from unittest.mock import Mock, patch
from tutor.__about__ import __version__
from tutor import config as tutor_config
from tutor import env, exceptions, fmt
from tutor.types import Config
from tests.helpers import temporary_root
class EnvTests(unittest.TestCase):
@ -30,9 +32,9 @@ class EnvTests(unittest.TestCase):
self.assertTrue(os.path.exists(path))
def test_pathjoin(self) -> None:
with tempfile.TemporaryDirectory() as root:
with temporary_root() as root:
self.assertEqual(
os.path.join(root, "env", "dummy"), env.pathjoin(root, "dummy")
os.path.join(env.base_dir(root), "dummy"), env.pathjoin(root, "dummy")
)
def test_render_str(self) -> None:
@ -76,21 +78,26 @@ class EnvTests(unittest.TestCase):
)
def test_save_full(self) -> None:
with tempfile.TemporaryDirectory() as root:
with temporary_root() as root:
config = tutor_config.load_full(root)
with patch.object(fmt, "STDOUT"):
env.save(root, config)
self.assertTrue(
os.path.exists(os.path.join(root, "env", "local", "docker-compose.yml"))
os.path.exists(
os.path.join(env.base_dir(root), "local", "docker-compose.yml")
)
)
def test_save_full_with_https(self) -> None:
with tempfile.TemporaryDirectory() as root:
with temporary_root() as root:
config = tutor_config.load_full(root)
config["ENABLE_HTTPS"] = True
with patch.object(fmt, "STDOUT"):
env.save(root, config)
with open(os.path.join(root, "env", "apps", "caddy", "Caddyfile")) as f:
with open(
os.path.join(env.base_dir(root), "apps", "caddy", "Caddyfile"),
encoding="utf-8",
) as f:
self.assertIn("www.myopenedx.com{$default_site_port}", f.read())
def test_patch(self) -> None:
@ -120,11 +127,15 @@ class EnvTests(unittest.TestCase):
# Create two templates
os.makedirs(os.path.join(plugin_templates, "plugin1", "apps"))
with open(
os.path.join(plugin_templates, "plugin1", "unrendered.txt"), "w"
os.path.join(plugin_templates, "plugin1", "unrendered.txt"),
"w",
encoding="utf-8",
) as f:
f.write("This file should not be rendered")
with open(
os.path.join(plugin_templates, "plugin1", "apps", "rendered.txt"), "w"
os.path.join(plugin_templates, "plugin1", "apps", "rendered.txt"),
"w",
encoding="utf-8",
) as f:
f.write("Hello my ID is {{ ID }}")
@ -137,7 +148,7 @@ class EnvTests(unittest.TestCase):
"iter_enabled",
return_value=[plugin1],
):
with tempfile.TemporaryDirectory() as root:
with temporary_root() as root:
# Render plugin templates
env.save_plugin_templates(plugin1, root, config)
@ -150,7 +161,7 @@ class EnvTests(unittest.TestCase):
)
self.assertFalse(os.path.exists(dst_unrendered))
self.assertTrue(os.path.exists(dst_rendered))
with open(dst_rendered) as f:
with open(dst_rendered, encoding="utf-8") as f:
self.assertEqual("Hello my ID is abcd", f.read())
def test_renderer_is_reset_on_config_change(self) -> None:
@ -161,7 +172,9 @@ class EnvTests(unittest.TestCase):
# Create one template
os.makedirs(os.path.join(plugin_templates, plugin1.name))
with open(
os.path.join(plugin_templates, plugin1.name, "myplugin.txt"), "w"
os.path.join(plugin_templates, plugin1.name, "myplugin.txt"),
"w",
encoding="utf-8",
) as f:
f.write("some content")
@ -199,3 +212,40 @@ class EnvTests(unittest.TestCase):
)
),
)
class CurrentVersionTests(unittest.TestCase):
def test_current_version_in_empty_env(self) -> None:
with temporary_root() as root:
self.assertIsNone(env.current_version(root))
self.assertIsNone(env.get_env_release(root))
self.assertIsNone(env.should_upgrade_from_release(root))
self.assertTrue(env.is_up_to_date(root))
def test_current_version_in_lilac_env(self) -> None:
with temporary_root() as root:
os.makedirs(env.base_dir(root))
with open(
os.path.join(env.base_dir(root), env.VERSION_FILENAME),
"w",
encoding="utf-8",
) as f:
f.write("12.0.46")
self.assertEqual("12.0.46", env.current_version(root))
self.assertEqual("lilac", env.get_env_release(root))
self.assertEqual("lilac", env.should_upgrade_from_release(root))
self.assertFalse(env.is_up_to_date(root))
def test_current_version_in_latest_env(self) -> None:
with temporary_root() as root:
os.makedirs(env.base_dir(root))
with open(
os.path.join(env.base_dir(root), env.VERSION_FILENAME),
"w",
encoding="utf-8",
) as f:
f.write(__version__)
self.assertEqual(__version__, env.current_version(root))
self.assertEqual("maple", env.get_env_release(root))
self.assertIsNone(env.should_upgrade_from_release(root))
self.assertTrue(env.is_up_to_date(root))

2
tutor/__about__.py
View File

@ -2,7 +2,7 @@ import os
# Increment this version number to trigger a new release. See
# docs/tutor.html#versioning for information on the versioning scheme.
__version__ = "13.0.3"
__version__ = "13.1.0"
# The version suffix will be appended to the actual version, separated by a
# dash. Use this suffix to differentiate between the actual released version and

231
tutor/commands/k8s.py
View File

@ -4,12 +4,13 @@ from typing import Any, List, Optional, Type
import click
from .. import config as tutor_config
from .. import env as tutor_env
from .. import exceptions, fmt, jobs, serialize, utils
from ..types import Config, get_typed
from .config import save as config_save_command
from .context import Context
from tutor import config as tutor_config
from tutor import env as tutor_env
from tutor import exceptions, fmt, jobs, serialize, utils
from tutor.commands.config import save as config_save_command
from tutor.commands.context import Context
from tutor.commands.upgrade.k8s import upgrade_from
from tutor.types import Config, get_typed
class K8sClients:
@ -50,11 +51,11 @@ class K8sJobRunner(jobs.BaseJobRunner):
job_name = job["metadata"]["name"]
if not isinstance(job_name, str):
raise exceptions.TutorError(
"Invalid job name: '{}'. Expected str.".format(job_name)
f"Invalid job name: '{job_name}'. Expected str."
)
if job_name == name:
return job
raise exceptions.TutorError("Could not find job '{}'".format(name))
raise exceptions.TutorError(f"Could not find job '{name}'")
def active_job_names(self) -> List[str]:
"""
@ -70,7 +71,7 @@ class K8sJobRunner(jobs.BaseJobRunner):
]
def run_job(self, service: str, command: str) -> int:
job_name = "{}-job".format(service)
job_name = f"{service}-job"
job = self.load_job(job_name)
# Create a unique job name to make it deduplicate jobs and make it easier to
# find later. Logs of older jobs will remain available for some time.
@ -82,7 +83,7 @@ class K8sJobRunner(jobs.BaseJobRunner):
if not active_jobs:
break
fmt.echo_info(
"Waiting for active jobs to terminate: {}".format(" ".join(active_jobs))
f"Waiting for active jobs to terminate: {' '.join(active_jobs)}"
)
sleep(5)
@ -105,7 +106,9 @@ class K8sJobRunner(jobs.BaseJobRunner):
job["spec"]["backoffLimit"] = 1
job["spec"]["ttlSecondsAfterFinished"] = 3600
# Save patched job to "jobs.yml" file
with open(tutor_env.pathjoin(self.root, "k8s", "jobs.yml"), "w") as job_file:
with open(
tutor_env.pathjoin(self.root, "k8s", "jobs.yml"), "w", encoding="utf-8"
) as job_file:
serialize.dump(job, job_file)
# We cannot use the k8s API to create the job: configMap and volume names need
# to be found with the right suffixes.
@ -114,7 +117,7 @@ class K8sJobRunner(jobs.BaseJobRunner):
"--kustomize",
tutor_env.pathjoin(self.root),
"--selector",
"app.kubernetes.io/name={}".format(job_name),
f"app.kubernetes.io/name={job_name}",
)
message = (
@ -126,7 +129,7 @@ class K8sJobRunner(jobs.BaseJobRunner):
fmt.echo_info(message)
# Wait for completion
field_selector = "metadata.name={}".format(job_name)
field_selector = f"metadata.name={job_name}"
while True:
namespaced_jobs = K8sClients.instance().batch_api.list_namespaced_job(
k8s_namespace(self.config), field_selector=field_selector
@ -136,13 +139,11 @@ class K8sJobRunner(jobs.BaseJobRunner):
job = namespaced_jobs.items[0]
if not job.status.active:
if job.status.succeeded:
fmt.echo_info("Job {} successful.".format(job_name))
fmt.echo_info(f"Job {job_name} successful.")
break
if job.status.failed:
raise exceptions.TutorError(
"Job {} failed. View the job logs to debug this issue.".format(
job_name
)
f"Job {job_name} failed. View the job logs to debug this issue."
)
sleep(5)
return 0
@ -157,30 +158,41 @@ def k8s() -> None:
@click.option("-I", "--non-interactive", is_flag=True, help="Run non-interactively")
@click.pass_context
def quickstart(context: click.Context, non_interactive: bool) -> None:
run_upgrade_from_release = tutor_env.should_upgrade_from_release(context.obj.root)
if run_upgrade_from_release is not None:
click.echo(fmt.title("Upgrading from an older release"))
context.invoke(
upgrade,
from_version=tutor_env.get_env_release(context.obj.root),
)
click.echo(fmt.title("Interactive platform configuration"))
context.invoke(
config_save_command,
interactive=(not non_interactive),
set_vars=[],
unset_vars=[],
)
config = tutor_config.load(context.obj.root)
if not config["ENABLE_WEB_PROXY"]:
fmt.echo_alert(
"Potentially invalid configuration: ENABLE_WEB_PROXY=false\n"
"This setting might have been defined because you previously set WEB_PROXY=true. This is no longer"
" necessary in order to get Tutor to work on Kubernetes. In Tutor v11+ a Caddy-based load balancer is"
" provided out of the box to handle SSL/TLS certificate generation at runtime. If you disable this"
" service, you will have to configure an Ingress resource and a certificate manager yourself to redirect"
" traffic to the caddy service. See the Kubernetes section in the Tutor documentation for more"
" information."
if run_upgrade_from_release and not non_interactive:
question = f"""Your platform is being upgraded from {run_upgrade_from_release.capitalize()}.
If you run custom Docker images, you must rebuild and push them to your private repository now by running the following
commands in a different shell:
tutor images build all # add your custom images here
tutor images push all
Press enter when you are ready to continue"""
click.confirm(
fmt.question(question), default=True, abort=True, prompt_suffix=" "
)
click.echo(fmt.title("Updating the current environment"))
tutor_env.save(context.obj.root, config)
click.echo(fmt.title("Starting the platform"))
context.invoke(start)
click.echo(fmt.title("Database creation and migrations"))
context.invoke(init, limit=None)
config = tutor_config.load(context.obj.root)
fmt.echo_info(
"""Your Open edX platform is ready and can be accessed at the following urls:
@ -248,7 +260,7 @@ def start(context: Context, names: List[str]) -> None:
"--kustomize",
tutor_env.pathjoin(context.root),
"--selector",
"app.kubernetes.io/name={}".format(name),
f"app.kubernetes.io/name={name}",
)
@ -264,25 +276,29 @@ def start(context: Context, names: List[str]) -> None:
def stop(context: Context, names: List[str]) -> None:
config = tutor_config.load(context.root)
names = names or ["all"]
resource_types = "deployments,services,configmaps,jobs"
not_lb_selector = "app.kubernetes.io/component!=loadbalancer"
for name in names:
if name == "all":
utils.kubectl(
"delete",
*resource_selector(config, not_lb_selector),
resource_types,
)
delete_resources(config)
else:
utils.kubectl(
"delete",
*resource_selector(
config,
not_lb_selector,
"app.kubernetes.io/name={}".format(name),
),
resource_types,
)
delete_resources(config, name=name)
def delete_resources(
config: Config, resources: Optional[List[str]] = None, name: Optional[str] = None
) -> None:
"""
Delete resources by type and name.
The load balancer is never deleted.
"""
resources = resources or ["deployments", "services", "configmaps", "jobs"]
not_lb_selector = "app.kubernetes.io/component!=loadbalancer"
name_selector = [f"app.kubernetes.io/name={name}"] if name else []
utils.kubectl(
"delete",
*resource_selector(config, not_lb_selector, *name_selector),
",".join(resources),
)
@click.command(help="Reboot an existing platform")
@ -336,8 +352,8 @@ def scale(context: Context, deployment: str, replicas: int) -> None:
*resource_namespace_selector(
config,
),
"--replicas={}".format(replicas),
"deployment/{}".format(deployment),
f"--replicas={replicas}",
f"deployment/{deployment}",
)
@ -434,112 +450,41 @@ def wait(context: Context, name: str) -> None:
wait_for_pod_ready(config, name)
@click.command(help="Upgrade from a previous Open edX named release")
@click.command(
short_help="Perform release-specific upgrade tasks",
help="Perform release-specific upgrade tasks. To perform a full upgrade remember to run `quickstart`.",
)
@click.option(
"--from",
"from_version",
default="koa",
"from_release",
type=click.Choice(["ironwood", "juniper", "koa", "lilac"]),
)
@click.pass_obj
def upgrade(context: Context, from_version: str) -> None:
config = tutor_config.load(context.root)
running_version = from_version
if running_version == "ironwood":
upgrade_from_ironwood(config)
running_version = "juniper"
if running_version == "juniper":
upgrade_from_juniper(config)
running_version = "koa"
if running_version == "koa":
upgrade_from_koa(config)
running_version = "lilac"
if running_version == "lilac":
# Nothing to do here
running_version = "maple"
def upgrade_from_ironwood(config: Config) -> None:
if not config["RUN_MONGODB"]:
fmt.echo_info(
"You are not running MongDB (RUN_MONGODB=false). It is your "
"responsibility to upgrade your MongoDb instance to v3.6. There is "
"nothing left to do to upgrade from Ironwood."
@click.pass_context
def upgrade(context: click.Context, from_release: Optional[str]) -> None:
if from_release is None:
from_release = tutor_env.get_env_release(context.obj.root)
if from_release is None:
fmt.echo_info("Your environment is already up-to-date")
else:
fmt.echo_alert(
"This command only performs a partial upgrade of your Open edX platform. "
"To perform a full upgrade, you should run `tutor k8s quickstart`."
)
return
message = """Automatic release upgrade is unsupported in Kubernetes. To upgrade from Ironwood, you should upgrade
your MongoDb cluster from v3.2 to v3.6. You should run something similar to:
# Upgrade from v3.2 to v3.4
tutor k8s stop
tutor config save --set DOCKER_IMAGE_MONGODB=mongo:3.4.24
tutor k8s start
tutor k8s exec mongodb mongo --eval 'db.adminCommand({ setFeatureCompatibilityVersion: "3.4" })'
# Upgrade from v3.4 to v3.6
tutor k8s stop
tutor config save --set DOCKER_IMAGE_MONGODB=mongo:3.6.18
tutor k8s start
tutor k8s exec mongodb mongo --eval 'db.adminCommand({ setFeatureCompatibilityVersion: "3.6" })'
tutor config save --unset DOCKER_IMAGE_MONGODB"""
fmt.echo_info(message)
def upgrade_from_juniper(config: Config) -> None:
if not config["RUN_MYSQL"]:
fmt.echo_info(
"You are not running MySQL (RUN_MYSQL=false). It is your "
"responsibility to upgrade your MySQL instance to v5.7. There is "
"nothing left to do to upgrade from Juniper."
)
return
message = """Automatic release upgrade is unsupported in Kubernetes. To upgrade from Juniper, you should upgrade
your MySQL database from v5.6 to v5.7. You should run something similar to:
tutor k8s start
tutor k8s exec mysql bash -e -c "mysql_upgrade \
-u $(tutor config printvalue MYSQL_ROOT_USERNAME) \
--password='$(tutor config printvalue MYSQL_ROOT_PASSWORD)'
"""
fmt.echo_info(message)
def upgrade_from_koa(config: Config) -> None:
if not config["RUN_MONGODB"]:
fmt.echo_info(
"You are not running MongDB (RUN_MONGODB=false). It is your "
"responsibility to upgrade your MongoDb instance to v4.0. There is "
"nothing left to do to upgrade to Lilac from Koa."
)
return
message = """Automatic release upgrade is unsupported in Kubernetes. To upgrade from Koa to Lilac, you should upgrade
your MongoDb cluster from v3.6 to v4.0. You should run something similar to:
tutor k8s stop
tutor config save --set DOCKER_IMAGE_MONGODB=mongo:4.0.25
tutor k8s start
tutor k8s exec mongodb mongo --eval 'db.adminCommand({ setFeatureCompatibilityVersion: "4.0" })'
tutor config save --unset DOCKER_IMAGE_MONGODB
"""
fmt.echo_info(message)
upgrade_from(context.obj, from_release)
# We update the environment to update the version
context.invoke(config_save_command)
def kubectl_exec(
config: Config, service: str, command: str, attach: bool = False
) -> int:
selector = "app.kubernetes.io/name={}".format(service)
selector = f"app.kubernetes.io/name={service}"
pods = K8sClients.instance().core_api.list_namespaced_pod(
namespace=k8s_namespace(config), label_selector=selector
)
if not pods.items:
raise exceptions.TutorError(
"Could not find an active pod for the {} service".format(service)
f"Could not find an active pod for the {service} service"
)
pod_name = pods.items[0].metadata.name
@ -560,10 +505,10 @@ def kubectl_exec(
def wait_for_pod_ready(config: Config, service: str) -> None:
fmt.echo_info("Waiting for a {} pod to be ready...".format(service))
fmt.echo_info(f"Waiting for a {service} pod to be ready...")
utils.kubectl(
"wait",
*resource_selector(config, "app.kubernetes.io/name={}".format(service)),
*resource_selector(config, f"app.kubernetes.io/name={service}"),
"--for=condition=ContainersReady",
"--timeout=600s",
"pod",

222
tutor/commands/local.py
View File

@ -1,13 +1,14 @@
from time import sleep
from typing import Optional
import click
from .. import config as tutor_config
from .. import env as tutor_env
from .. import exceptions, fmt, utils
from ..types import Config, get_typed
from . import compose
from .config import save as config_save_command
from tutor import config as tutor_config
from tutor import env as tutor_env
from tutor import exceptions, fmt, utils
from tutor.commands import compose
from tutor.commands.config import save as config_save_command
from tutor.commands.upgrade.local import upgrade_from
from tutor.types import Config, get_typed
class LocalJobRunner(compose.ComposeJobRunner):
@ -28,6 +29,7 @@ class LocalJobRunner(compose.ComposeJobRunner):
]
# pylint: disable=too-few-public-methods
class LocalContext(compose.BaseComposeContext):
def job_runner(self, config: Config) -> LocalJobRunner:
return LocalJobRunner(self.root, config)
@ -49,27 +51,57 @@ def quickstart(context: click.Context, non_interactive: bool, pullimages: bool)
except exceptions.TutorError as e:
fmt.echo_alert(
f"""Could not verify sufficient RAM allocation in Docker:
{e}
Tutor may not work if Docker is configured with < 4 GB RAM. Please follow the instructions from:
https://docs.tutor.overhang.io/install.html
"""
Tutor may not work if Docker is configured with < 4 GB RAM. Please follow instructions from:
https://docs.tutor.overhang.io/install.html"""
)
if tutor_env.needs_major_upgrade(context.obj.root):
run_upgrade_from_release = tutor_env.should_upgrade_from_release(context.obj.root)
if run_upgrade_from_release is not None:
click.echo(fmt.title("Upgrading from an older release"))
if not non_interactive:
to_release = tutor_env.get_package_release()
question = f"""You are about to upgrade your Open edX platform from {run_upgrade_from_release.capitalize()} to {to_release.capitalize()}
It is strongly recommended to make a backup before upgrading. To do so, run:
tutor local stop
sudo rsync -avr "$(tutor config printroot)"/ /tmp/tutor-backup/
In case of problem, to restore your backup you will then have to run: sudo rsync -avr /tmp/tutor-backup/ "$(tutor config printroot)"/
Are you sure you want to continue?"""
click.confirm(
fmt.question(question), default=True, abort=True, prompt_suffix=" "
)
context.invoke(
upgrade,
from_version=tutor_env.current_release(context.obj.root),
from_version=run_upgrade_from_release,
non_interactive=non_interactive,
)
click.echo(fmt.title("Interactive platform configuration"))
context.invoke(
config_save_command,
interactive=(not non_interactive),
set_vars=[],
unset_vars=[],
)
context.invoke(config_save_command, interactive=(not non_interactive))
if run_upgrade_from_release and not non_interactive:
question = f"""Your platform is being upgraded from {run_upgrade_from_release.capitalize()}.
If you run custom Docker images, you must rebuild them now by running the following command in a different shell:
tutor images build all # list your custom images here
See the documentation for more information:
https://docs.tutor.overhang.io/install.html#upgrading-to-a-new-open-edx-release
Press enter when you are ready to continue"""
click.confirm(
fmt.question(question), default=True, abort=True, prompt_suffix=" "
)
click.echo(fmt.title("Stopping any existing platform"))
context.invoke(compose.stop)
if pullimages:
@ -95,149 +127,29 @@ Your Open edX platform is ready and can be accessed at the following urls:
)
@click.command(help="Upgrade from a previous Open edX named release")
@click.command(
short_help="Perform release-specific upgrade tasks",
help="Perform release-specific upgrade tasks. To perform a full upgrade remember to run `quickstart`.",
)
@click.option(
"--from",
"from_version",
default="koa",
"from_release",
type=click.Choice(["ironwood", "juniper", "koa", "lilac"]),
)
@click.option("-I", "--non-interactive", is_flag=True, help="Run non-interactively")
@click.pass_context
def upgrade(context: click.Context, from_version: str, non_interactive: bool) -> None:
config = tutor_config.load_full(context.obj.root)
if not non_interactive:
question = """You are about to upgrade your Open edX platform. It is strongly recommended to make a backup before upgrading. To do so, run:
tutor local stop
sudo rsync -avr "$(tutor config printroot)"/ /tmp/tutor-backup/
In case of problem, to restore your backup you will then have to run: sudo rsync -avr /tmp/tutor-backup/ "$(tutor config printroot)"/
Are you sure you want to continue?"""
click.confirm(
fmt.question(question), default=True, abort=True, prompt_suffix=" "
)
running_version = from_version
if running_version == "ironwood":
upgrade_from_ironwood(context, config)
running_version = "juniper"
if running_version == "juniper":
upgrade_from_juniper(context, config)
running_version = "koa"
if running_version == "koa":
upgrade_from_koa(context, config)
running_version = "lilac"
if running_version == "lilac":
# Nothing to do here
running_version = "maple"
if not non_interactive:
question = f"""
Your platform was successfuly upgraded from {from_version} to {running_version}. Depending on your setup, you might have to rebuild some of your Docker images. You can do this now by running the following command in a different shell:
tutor images build openedx # add your custom images here
Press enter when you are ready to continue"""
click.confirm(
fmt.question(question), default=True, abort=True, prompt_suffix=" "
)
def upgrade_from_ironwood(context: click.Context, config: Config) -> None:
click.echo(fmt.title("Upgrading from Ironwood"))
tutor_env.save(context.obj.root, config)
click.echo(fmt.title("Stopping any existing platform"))
context.invoke(compose.stop)
if not config["RUN_MONGODB"]:
fmt.echo_info(
"You are not running MongDB (RUN_MONGODB=false). It is your "
"responsibility to upgrade your MongoDb instance to v3.6. There is "
"nothing left to do to upgrade from Ironwood to Juniper."
)
return
upgrade_mongodb(context, config, "3.4", "3.4")
context.invoke(compose.stop)
upgrade_mongodb(context, config, "3.6", "3.6")
context.invoke(compose.stop)
def upgrade_from_juniper(context: click.Context, config: Config) -> None:
click.echo(fmt.title("Upgrading from Juniper"))
tutor_env.save(context.obj.root, config)
click.echo(fmt.title("Stopping any existing platform"))
context.invoke(compose.stop)
if not config["RUN_MYSQL"]:
fmt.echo_info(
"You are not running MySQL (RUN_MYSQL=false). It is your "
"responsibility to upgrade your MySQL instance to v5.7. There is "
"nothing left to do to upgrade from Juniper."
)
return
click.echo(fmt.title("Upgrading MySQL from v5.6 to v5.7"))
context.invoke(compose.start, detach=True, services=["mysql"])
context.invoke(
compose.execute,
args=[
"mysql",
"bash",
"-e",
"-c",
"mysql_upgrade -u {} --password='{}'".format(
config["MYSQL_ROOT_USERNAME"], config["MYSQL_ROOT_PASSWORD"]
),
],
def upgrade(context: click.Context, from_release: Optional[str]) -> None:
fmt.echo_alert(
"This command only performs a partial upgrade of your Open edX platform. "
"To perform a full upgrade, you should run `tutor local quickstart`."
)
context.invoke(compose.stop)
def upgrade_from_koa(context: click.Context, config: Config) -> None:
if not config["RUN_MONGODB"]:
fmt.echo_info(
"You are not running MongDB (RUN_MONGODB=false). It is your "
"responsibility to upgrade your MongoDb instance to v4.0. There is "
"nothing left to do to upgrade from Koa to Lilac."
)
return
upgrade_mongodb(context, config, "4.0.25", "4.0")
def upgrade_mongodb(
context: click.Context,
config: Config,
to_docker_version: str,
to_compatibility_version: str,
) -> None:
click.echo(fmt.title("Upgrading MongoDb to v{}".format(to_docker_version)))
# Note that the DOCKER_IMAGE_MONGODB value is never saved, because we only save the
# environment, not the configuration.
config["DOCKER_IMAGE_MONGODB"] = "mongo:{}".format(to_docker_version)
tutor_env.save(context.obj.root, config)
context.invoke(compose.start, detach=True, services=["mongodb"])
fmt.echo_info("Waiting for mongodb to boot...")
sleep(10)
context.invoke(
compose.execute,
args=[
"mongodb",
"mongo",
"--eval",
'db.adminCommand({ setFeatureCompatibilityVersion: "%s" })'
% to_compatibility_version,
],
)
context.invoke(compose.stop)
if from_release is None:
from_release = tutor_env.get_env_release(context.obj.root)
if from_release is None:
fmt.echo_info("Your environment is already up-to-date")
else:
upgrade_from(context, from_release)
# We update the environment to update the version
context.invoke(config_save_command)
local.add_command(quickstart)

0
tutor/commands/upgrade/__init__.py Normal file
View File

31
tutor/commands/upgrade/common.py Normal file
View File

@ -0,0 +1,31 @@
from tutor import fmt
from tutor import plugins
from tutor.types import Config
def upgrade_from_lilac(config: Config) -> None:
if not plugins.is_installed("forum"):
fmt.echo_alert(
"The Open edX forum feature was moved to a separate plugin in Maple. To keep using this feature, "
"you must install and enable the tutor-forum plugin: https://github.com/overhangio/tutor-forum"
)
elif not plugins.is_enabled(config, "forum"):
fmt.echo_info(
"The Open edX forum feature was moved to a separate plugin in Maple. To keep using this feature, "
"we will now enable the 'forum' plugin. If you do not want to use this feature, you should disable the "
"plugin with: `tutor plugins disable forum`."
)
plugins.enable(config, "forum")
if not plugins.is_installed("mfe"):
fmt.echo_alert(
"In Maple the legacy courseware is no longer supported. You need to install and enable the 'mfe' plugin "
"to make use of the new learning microfrontend: https://github.com/overhangio/tutor-mfe"
)
elif not plugins.is_enabled(config, "mfe"):
fmt.echo_info(
"In Maple the legacy courseware is no longer supported. To start using the new learning microfrontend, "
"we will now enable the 'mfe' plugin. If you do not want to use this feature, you should disable the "
"plugin with: `tutor plugins disable mfe`."
)
plugins.enable(config, "mfe")

103
tutor/commands/upgrade/k8s.py Normal file
View File

@ -0,0 +1,103 @@
from tutor import config as tutor_config
from tutor import fmt
from tutor.commands import k8s
from tutor.commands.context import Context
from tutor.types import Config
from . import common as common_upgrade
def upgrade_from(context: Context, from_release: str) -> None:
config = tutor_config.load(context.root)
running_release = from_release
if running_release == "ironwood":
upgrade_from_ironwood(config)
running_release = "juniper"
if running_release == "juniper":
upgrade_from_juniper(config)
running_release = "koa"
if running_release == "koa":
upgrade_from_koa(config)
running_release = "lilac"
if running_release == "lilac":
upgrade_from_lilac(config)
running_release = "maple"
def upgrade_from_ironwood(config: Config) -> None:
if not config["RUN_MONGODB"]:
fmt.echo_info(
"You are not running MongDB (RUN_MONGODB=false). It is your "
"responsibility to upgrade your MongoDb instance to v3.6. There is "
"nothing left to do to upgrade from Ironwood."
)
return
message = """Automatic release upgrade is unsupported in Kubernetes. To upgrade from Ironwood, you should upgrade
your MongoDb cluster from v3.2 to v3.6. You should run something similar to:
# Upgrade from v3.2 to v3.4
tutor k8s stop
tutor config save --set DOCKER_IMAGE_MONGODB=mongo:3.4.24
tutor k8s start
tutor k8s exec mongodb mongo --eval 'db.adminCommand({ setFeatureCompatibilityVersion: "3.4" })'
# Upgrade from v3.4 to v3.6
tutor k8s stop
tutor config save --set DOCKER_IMAGE_MONGODB=mongo:3.6.18
tutor k8s start
tutor k8s exec mongodb mongo --eval 'db.adminCommand({ setFeatureCompatibilityVersion: "3.6" })'
tutor config save --unset DOCKER_IMAGE_MONGODB"""
fmt.echo_info(message)
def upgrade_from_juniper(config: Config) -> None:
if not config["RUN_MYSQL"]:
fmt.echo_info(
"You are not running MySQL (RUN_MYSQL=false). It is your "
"responsibility to upgrade your MySQL instance to v5.7. There is "
"nothing left to do to upgrade from Juniper."
)
return
message = """Automatic release upgrade is unsupported in Kubernetes. To upgrade from Juniper, you should upgrade
your MySQL database from v5.6 to v5.7. You should run something similar to:
tutor k8s start
tutor k8s exec mysql bash -e -c "mysql_upgrade \
-u $(tutor config printvalue MYSQL_ROOT_USERNAME) \
--password='$(tutor config printvalue MYSQL_ROOT_PASSWORD)'
"""
fmt.echo_info(message)
def upgrade_from_koa(config: Config) -> None:
if not config["RUN_MONGODB"]:
fmt.echo_info(
"You are not running MongDB (RUN_MONGODB=false). It is your "
"responsibility to upgrade your MongoDb instance to v4.0. There is "
"nothing left to do to upgrade to Lilac from Koa."
)
return
message = """Automatic release upgrade is unsupported in Kubernetes. To upgrade from Koa to Lilac, you should upgrade
your MongoDb cluster from v3.6 to v4.0. You should run something similar to:
tutor k8s stop
tutor config save --set DOCKER_IMAGE_MONGODB=mongo:4.0.25
tutor k8s start
tutor k8s exec mongodb mongo --eval 'db.adminCommand({ setFeatureCompatibilityVersion: "4.0" })'
tutor config save --unset DOCKER_IMAGE_MONGODB
"""
fmt.echo_info(message)
def upgrade_from_lilac(config: Config) -> None:
common_upgrade.upgrade_from_lilac(config)
fmt.echo_info(
"All Kubernetes services and deployments need to be deleted during "
"upgrade from Lilac to Maple"
)
k8s.delete_resources(config, resources=["deployments", "services"])

120
tutor/commands/upgrade/local.py Normal file
View File

@ -0,0 +1,120 @@
from time import sleep
import click
from tutor import config as tutor_config
from tutor import env as tutor_env
from tutor import fmt
from tutor.commands import compose
from tutor.types import Config
from . import common as common_upgrade
def upgrade_from(context: click.Context, from_release: str) -> None:
# Make sure to bypass current version check
config = tutor_config.load_full(context.obj.root)
running_release = from_release
if running_release == "ironwood":
upgrade_from_ironwood(context, config)
running_release = "juniper"
if running_release == "juniper":
upgrade_from_juniper(context, config)
running_release = "koa"
if running_release == "koa":
upgrade_from_koa(context, config)
running_release = "lilac"
if running_release == "lilac":
common_upgrade.upgrade_from_lilac(config)
running_release = "maple"
def upgrade_from_ironwood(context: click.Context, config: Config) -> None:
click.echo(fmt.title("Upgrading from Ironwood"))
tutor_env.save(context.obj.root, config)
click.echo(fmt.title("Stopping any existing platform"))
context.invoke(compose.stop)
if not config["RUN_MONGODB"]:
fmt.echo_info(
"You are not running MongDB (RUN_MONGODB=false). It is your "
"responsibility to upgrade your MongoDb instance to v3.6. There is "
"nothing left to do to upgrade from Ironwood to Juniper."
)
return
upgrade_mongodb(context, config, "3.4", "3.4")
context.invoke(compose.stop)
upgrade_mongodb(context, config, "3.6", "3.6")
context.invoke(compose.stop)
def upgrade_from_juniper(context: click.Context, config: Config) -> None:
click.echo(fmt.title("Upgrading from Juniper"))
tutor_env.save(context.obj.root, config)
click.echo(fmt.title("Stopping any existing platform"))
context.invoke(compose.stop)
if not config["RUN_MYSQL"]:
fmt.echo_info(
"You are not running MySQL (RUN_MYSQL=false). It is your "
"responsibility to upgrade your MySQL instance to v5.7. There is "
"nothing left to do to upgrade from Juniper."
)
return
click.echo(fmt.title("Upgrading MySQL from v5.6 to v5.7"))
context.invoke(compose.start, detach=True, services=["mysql"])
context.invoke(
compose.execute,
args=[
"mysql",
"bash",
"-e",
"-c",
f"mysql_upgrade -u {config['MYSQL_ROOT_USERNAME']} --password='{config['MYSQL_ROOT_PASSWORD']}'",
],
)
context.invoke(compose.stop)
def upgrade_from_koa(context: click.Context, config: Config) -> None:
click.echo(fmt.title("Upgrading from Koa"))
if not config["RUN_MONGODB"]:
fmt.echo_info(
"You are not running MongDB (RUN_MONGODB=false). It is your "
"responsibility to upgrade your MongoDb instance to v4.0. There is "
"nothing left to do to upgrade from Koa to Lilac."
)
return
upgrade_mongodb(context, config, "4.0.25", "4.0")
def upgrade_mongodb(
context: click.Context,
config: Config,
to_docker_version: str,
to_compatibility_version: str,
) -> None:
click.echo(fmt.title(f"Upgrading MongoDb to v{to_docker_version}"))
# Note that the DOCKER_IMAGE_MONGODB value is never saved, because we only save the
# environment, not the configuration.
config["DOCKER_IMAGE_MONGODB"] = f"mongo:{to_docker_version}"
tutor_env.save(context.obj.root, config)
context.invoke(compose.start, detach=True, services=["mongodb"])
fmt.echo_info("Waiting for mongodb to boot...")
sleep(10)
context.invoke(
compose.execute,
args=[
"mongodb",
"mongo",
"--eval",
f'db.adminCommand({{ setFeatureCompatibilityVersion: "{to_compatibility_version}" }})',
],
)
context.invoke(compose.stop)

79
tutor/env.py
View File

@ -1,4 +1,3 @@
import codecs
import os
from copy import deepcopy
from typing import Any, Iterable, List, Optional, Type, Union
@ -139,9 +138,7 @@ class Renderer:
try:
patches.append(self.render_str(patch))
except exceptions.TutorError:
fmt.echo_error(
"Error rendering patch '{}' from plugin {}".format(name, plugin)
)
fmt.echo_error(f"Error rendering patch '{name}' from plugin {plugin}")
raise
rendered = separator.join(patches)
if rendered:
@ -193,9 +190,7 @@ class Renderer:
try:
return template.render(**self.config)
except jinja2.exceptions.UndefinedError as e:
raise exceptions.TutorError(
"Missing configuration value: {}".format(e.args[0])
)
raise exceptions.TutorError(f"Missing configuration value: {e.args[0]}")
def save(root: str, config: Config) -> None:
@ -219,7 +214,7 @@ def save(root: str, config: Config) -> None:
save_plugin_templates(plugin, root, config)
upgrade_obsolete(root)
fmt.echo_info("Environment generated in {}".format(base_dir(root)))
fmt.echo_info(f"Environment generated in {base_dir(root)}")
def upgrade_obsolete(_root: str) -> None:
@ -280,7 +275,7 @@ def render_unknown(config: Config, value: Any) -> Any:
"""
if isinstance(value, str):
return render_str(config, value)
elif isinstance(value, dict):
if isinstance(value, dict):
return {k: render_unknown(config, v) for k, v in value.items()}
return value
@ -299,15 +294,12 @@ def render_str(config: Config, text: str) -> str:
def check_is_up_to_date(root: str) -> None:
if not is_up_to_date(root):
message = (
"The current environment stored at {} is not up-to-date: it is at "
"v{} while the 'tutor' binary is at v{}. You should upgrade "
"the environment by running:\n"
"\n"
" tutor config save"
)
fmt.echo_alert(
message.format(base_dir(root), current_version(root), __version__)
f"The current environment stored at {base_dir(root)} is not up-to-date: it is at "
f"v{current_version(root)} while the 'tutor' binary is at v{__version__}. You should upgrade "
f"the environment by running:\n"
f"\n"
f" tutor config save"
)
@ -315,22 +307,45 @@ def is_up_to_date(root: str) -> bool:
"""
Check if the currently rendered version is equal to the current tutor version.
"""
return current_version(root) == __version__
current = current_version(root)
return current is None or current == __version__
def needs_major_upgrade(root: str) -> bool:
def should_upgrade_from_release(root: str) -> Optional[str]:
"""
Return the current version as a tuple of int. E.g: (1, 0, 2).
Return the name of the currently installed release that we should upgrade from. Return None If we already run the
latest release.
"""
current = int(current_version(root).split(".")[0])
required = int(__version__.split(".")[0])
return 0 < current < required
current = current_version(root)
if current is None:
return None
current_as_int = int(current.split(".")[0])
required_as_int = int(__version__.split(".", maxsplit=1)[0])
if current_as_int >= required_as_int:
return None
return get_release(current)
def current_release(root: str) -> str:
def get_env_release(root: str) -> Optional[str]:
"""
Return the name of the current Open edX release.
Return the Open edX release name from the current environment.
If the current environment has no version, return None.
"""
version = current_version(root)
if version is None:
return None
return get_release(version)
def get_package_release() -> str:
"""
Return the release name associated to this package.
"""
return get_release(__version__)
def get_release(version: str) -> str:
return {
"0": "ironwood",
"3": "ironwood",
@ -338,19 +353,19 @@ def current_release(root: str) -> str:
"11": "koa",
"12": "lilac",
"13": "maple",
}[current_version(root).split(".")[0]]
}[version.split(".", maxsplit=1)[0]]
def current_version(root: str) -> str:
def current_version(root: str) -> Optional[str]:
"""
Return the current environment version. If the current environment has no version,
return "0.0.0".
return None.
"""
path = pathjoin(root, VERSION_FILENAME)
if not os.path.exists(path):
return "0.0.0"
with open(path) as f:
return f.read().strip()
return None
with open(path, encoding="utf-8") as fi:
return fi.read().strip()
def read_template_file(*path: str) -> str:
@ -358,7 +373,7 @@ def read_template_file(*path: str) -> str:
Read raw content of template located at `path`.
"""
src = template_path(*path)
with codecs.open(src, encoding="utf-8") as fi:
with open(src, encoding="utf-8") as fi:
return fi.read()

6
tutor/templates/k8s/deployments.yml
View File

@ -340,9 +340,9 @@ spec:
app.kubernetes.io/name: mysql
spec:
securityContext:
runAsUser: 1000
runAsGroup: 1000
fsGroup: 1000
runAsUser: 999
runAsGroup: 999
fsGroup: 999
fsGroupChangePolicy: "OnRootMismatch"
containers:
- name: mysql

4
tutor/templates/local/docker-compose.yml
View File

@ -28,7 +28,7 @@ services:
image: {{ DOCKER_IMAGE_MYSQL }}
command: mysqld --character-set-server=utf8 --collation-server=utf8_general_ci
restart: unless-stopped
user: "1000:1000"
user: "999:999"
privileged: false
volumes:
- ../../data/mysql:/var/lib/mysql
@ -36,7 +36,7 @@ services:
MYSQL_ROOT_PASSWORD: "{{ MYSQL_ROOT_PASSWORD }}"
mysql-permissions:
image: {{ DOCKER_IMAGE_PERMISSIONS }}
command: ["1000", "/var/lib/mysql"]
command: ["999", "/var/lib/mysql"]
restart: on-failure
volumes:
- ../../data/mysql:/var/lib/mysql