Merge remote-tracking branch 'origin/master' into nightly

changelog.d/20230103_165634_regis_plugin_indices.md

@@ -0,0 +1,3 @@
+- [Feature] Introduce plugin indexes, described in this [Tutor enhancement proposal](https://discuss.openedx.org/t/tutor-enhancement-proposal-tep-plugin-indices/8182). This new feature introduces a lot of new ``plugins`` commands. See the docs for more information. (by @regisb)
+- [Improvement] Add the `plugins list --enabled` option. (by @regisb)
+- 💥[Improvement] Modify the output of `plugins list`. Enabled plugins are indicated as "enabled". Installed but not enabled plugins are no longer indicated as "disabled" but as "installed".

docs/plugins/intro.rst

@@ -4,7 +4,7 @@
 Introduction
 ============
 
-Tutor comes with a plugin system that allows anyone to customise the deployment of an Open edX platform very easily. The vision behind this plugin system is that users should not have to fork the Tutor repository to customise their deployments. For instance, if you have created a new application that integrates with Open edX, you should not have to describe how to manually patch the platform settings, ``urls.py`` or ``*.env.yml`` files. Instead, you can create a "tutor-myapp" plugin for Tutor. Then, users will start using your application in three simple steps::
+Tutor comes with a plugin system that allows anyone to customise the deployment of an Open edX platform very easily. The vision behind this plugin system is that users should not have to fork the Tutor repository to customise their deployments. For instance, if you have created a new application that integrates with Open edX, you should not have to describe how to manually patch the platform settings, ``urls.py`` or ``*.env.yml`` files. Instead, you can create a "tutor-myapp" plugin for Tutor. This plugin will be in charge of making changes to the platform settings. Then, users will be able to use your application in three simple steps::
 
     # 1) Install the plugin
     pip install tutor-myapp
@@ -40,4 +40,29 @@ The full plugins CLI is described in the :ref:`reference documentation <cli_plug
 Existing plugins
 ================
 
-Officially-supported plugins are listed on the `Overhang.IO <https://overhang.io/tutor/plugins>`__ website.
+Many plugins are available from plugin indexes. These indexes are lists of plugins, similar to the `pypi <https://pypi.org>`__ or `npm <npmjs.com/>`__ indexes. By default, Tutor comes with the "main" plugin index. You can check available plugins from this index by running::
+
+    tutor plugins update
+    tutor plugins search
+
+More plugins can be downloaded from the "contrib" and "wizard" indexes::
+
+    tutor plugins index add contrib
+    tutor plugins index add wizard
+    tutor plugins search
+
+The "main", "contrib" and "wizard" indexes include a curated list of plugins that are well maintained and introduce useful features to Open edX. These indexes are maintained by `Overhang.IO <https://overhang.io>`__. For more information about these indexes, refer to the official `overhangio/tpi <https://github.com/overhangio/tpi>`__ repository.
+
+Thanks to these indexes, it is very easy to download and upgrade plugins. For instance, to install the `notes plugin <https://github.com/overhangio/tutor-notes/>`__::
+
+    tutor plugins install notes
+
+Upgrade all your plugins with::
+
+    tutor plugins upgrade all
+
+To list indexes that you are downloading plugins from, run::
+
+    tutor plugins index list
+
+For more information about these indexes, check the `official Tutor plugin indexes (TPI) <https://github.com/overhangio/tpi/>`__ repository.

docs/reference/index.rst

@@ -8,3 +8,4 @@ Reference
    api/hooks/catalog
    patches
    cli/index
+   indexes

docs/reference/indexes.rst

@@ -0,0 +1,158 @@
+==============
+Plugin indexes
+==============
+
+Plugin indexes are a great way to have your plugins discovered by other users. Plugin indexes make it easy for other Tutor users to install and upgrade plugins from other developers. Examples include the official indexes, which can be found in the `overhangio/tpi <https://github.com/overhangio/tpi/>`__ repository.
+
+Index file paths
+================
+
+A plugin index is a yaml-formatted file. It can be stored on the web or on your computer. In both cases, the index file location must end with "<current release name>/plugins.yml". For instance, the following are valid index locations if you run the Open edX "Olive" release:
+
+- https://overhang.io/tutor/main/olive/plugins.yml
+- ``/path/to/your/local/index/olive/plugins.yml``
+
+To add either indexes, run the ``tutor plugins index add`` command without the suffix. For instance::
+
+    tutor plugins index add https://overhang.io/tutor/main
+    tutor plugins index add /path/to/your/local/index/
+
+Your plugin cache should be updated immediately. You can also update the cache at any point by running::
+
+    tutor plugins update
+
+To view current indexes, run::
+
+    tutor plugins index list
+
+To remove an index, run::
+
+    tutor plugins index remove <index url>
+
+Plugin entry syntax
+===================
+
+A "plugins.yml" file is a yaml-formatted list of plugin entries. Each plugin entry has two required fields: "name" and "src". For instance, here is a minimal plugin entry::
+
+    - name: mfe
+      src: tutor-mfe
+
+"name" (required)
+-----------------
+
+A plugin name is how it will be referenced when we run ``tutor plugins install <name>`` or ``tutor plugins enable <name>``. It should be concise and easily identifiable, just like a Python or apt package name.
+
+Plugins with duplicate names will be overridden, depending on the index in which they are declared: indexes further down ``tutor plugins index list`` (which have been added later) will have higher priority.
+
+.. _plugin_index_src:
+
+"src" (required)
+----------------
+
+A plugin source can be either:
+
+1. A pip requirement file format specifier (see `reference <https://pip.pypa.io/en/stable/reference/requirements-file-format/>`__).
+2. The path to a Python file on your computer.
+3. The URL of a Python file on the web.
+
+In the first case, the plugin will be installed as a Python package. In the other two cases, the plugin will be installed as a single-file plugin.
+
+The following "src" attributes are all valid::
+
+    # Pypi package
+    src: tutor-mfe
+
+    # Pypi package with version specification
+    src: tutor-mfe>=42.0.0,<43.0.0
+
+    # Python package from a private index
+    src: |
+        --index-url=https://pip.mymirror.org
+        my-plugin>=10.0
+
+    # Remote git repository
+    src: -e git+https://github.com/myusername/tutor-contrib-myplugin@v27.0.0#egg=tutor-contrib-myplugin
+
+    # Local editable package
+    src: -e /path/to/my/plugin
+
+"url" (optional)
+----------------
+
+Link to the plugin project, where users can learn more about it and ask for support.
+
+"author" (optional)
+-------------------
+
+Original author of the plugin. Feel free to include your company name and email address here. For instance: "Leather Face <niceguy@happyfamily.com>".
+
+"maintainer" (optional)
+-----------------------
+
+Current maintainer of the plugin. Same format as "author".
+
+"description" (optional)
+------------------------
+
+Multi-line string that should contain extensive information about your plugin. The full description will be displayed with ``tutor plugins show <name>``. It will also be parsed for a match by ``tutor plugins search <pattern>``. Only the first line will be displayed in the output of ``tutor plugins search``. Make sure to keep the first line below 128 characters.
+
+
+Examples
+========
+
+Manage plugins in development
+-----------------------------
+
+Plugin developers and maintainers often want to install local versions of their plugins. They usually achieve this with ``pip install -e /path/to/tutor-plugin``. We can improve that workflow by creating an index for local plugins::
+
+    # Create the plugin index directory
+    mkdir -p ~/localindex/olive/
+    # Edit the index
+    vim ~/localindex/olive/plugins.yml
+
+Add the following to the index::
+
+    - name: myplugin1
+      src: -e /path/to/tutor-myplugin1
+    - name: myplugin2
+      src: -e /path/to/tutor-myplugin2
+
+Then add the index::
+
+    tutor plugins index add ~/localindex/
+
+Install the plugins::
+
+    tutor plugins install myplugin1 myplugin2
+
+Re-install all plugins::
+
+    tutor plugins upgrade all
+
+The latter commands will install from the local index, and not from the remote indexes, because indexes that are added last have higher priority when plugins with the same names are found.
+
+Install plugins from a private index
+------------------------------------
+
+Plugin authors might want to share plugins with a limited number of users. This is for instance the case when a plugin is for internal use only.
+
+First, users should have access to the ``plugins.yml`` file. There are different ways to achieve that:
+
+- Make the index public: after all, it's mostly the plugins which are private. 
+- Grant access to the index from behind a VPN.
+- Hide the index behing a basic HTTP auth url. The index can then be added with ``tutor plugins index add http://user:password@mycompany.com/index/``.
+- Download the index to disk, and then add it from the local path: ``tutor plugins index add ../path/to/index``.
+
+Second, users should be able to install the plugins that are listed in the index. We recommend that the plugins are uploaded to a pip-compatible self-hosted mirror, such as `devpi <https://devpi.net/docs/devpi/devpi/latest/+doc/index.html>`__. Alternatively, packages can be installed from a private Git repository. For instance::
+
+    # Install from private pip index
+    - name: myprivateplugin1
+      src: |
+        --index-url=https://my-pip-index.mycompany.com/
+        tutor-contrib-myprivateplugin
+
+    # Install from private git repository
+    - name: myprivateplugin2
+      src: -e git+https://git.mycompany.com/tutor-contrib-myplugin2.git
+
+Both examples work because the :ref:`"src" <plugin_index_src>` field supports just any syntax that could also be included in a requirements file installed with ``pip install -r requirements.txt``.

tests/commands/test_plugins.py

@@ -1,3 +1,4 @@
+import typing as t
 import unittest
 from unittest.mock import Mock, patch
 
@@ -24,7 +25,7 @@ class PluginsTests(unittest.TestCase, TestCommandMixin):
         result = self.invoke(["plugins", "list"])
         self.assertIsNone(result.exception)
         self.assertEqual(0, result.exit_code)
-        self.assertFalse(result.output)
+        self.assertEqual("NAME\tSTATUS\tVERSION\n", result.output)
         _iter_info.assert_called()
 
     def test_plugins_install_not_found_plugin(self) -> None:
@@ -55,3 +56,17 @@ class PluginsTests(unittest.TestCase, TestCommandMixin):
         self.assertEqual(
             ["all", "alba"], plugins_commands.PluginName(allow_all=True).get_names("al")
         )
+
+    def test_format_table(self) -> None:
+        rows: t.List[t.Tuple[str, ...]] = [
+            ("a", "xyz", "value 1"),
+            ("abc", "x", "value 12345"),
+        ]
+        formatted = plugins_commands.format_table(rows, separator="  ")
+        self.assertEqual(
+            """
+a    xyz  value 1
+abc  x    value 12345
+""".strip(),
+            formatted,
+        )

tests/test_plugin_indexes.py

@@ -0,0 +1,81 @@
+import os
+import unittest
+from unittest.mock import patch
+
+from tutor.exceptions import TutorError
+from tutor.plugins import indexes
+from tutor.types import Config
+
+
+class PluginIndexesTest(unittest.TestCase):
+    def test_named_index_url(self) -> None:
+        self.assertEqual(
+            f"https://myindex.com/tutor/{indexes.RELEASE}/plugins.yml",
+            indexes.named_index_url("https://myindex.com/tutor"),
+        )
+        self.assertEqual(
+            f"https://myindex.com/tutor/{indexes.RELEASE}/plugins.yml",
+            indexes.named_index_url("https://myindex.com/tutor/"),
+        )
+
+        local_url = os.path.join("path", "to", "index", indexes.RELEASE)
+        self.assertEqual(
+            os.path.join(local_url, indexes.RELEASE, "plugins.yml"),
+            indexes.named_index_url(local_url),
+        )
+
+    def test_parse_index(self) -> None:
+        # Valid, empty index
+        self.assertEqual([], indexes.parse_index("[]"))
+        # Invalid index, list expected
+        with self.assertRaises(TutorError):
+            self.assertEqual([], indexes.parse_index("{}"))
+        # Invalid, empty index
+        with self.assertRaises(TutorError):
+            self.assertEqual([], indexes.parse_index("["))
+        # Partially valid index
+        with patch.object(indexes.fmt, "echo"):
+            self.assertEqual(
+                [{"name": "valid1"}],
+                indexes.parse_index(
+                    """
+- namE: invalid1
+- name: valid1
+        """
+                ),
+            )
+
+    def test_add(self) -> None:
+        config: Config = {}
+        self.assertTrue(indexes.add("https://myindex.com", config))
+        self.assertFalse(indexes.add("https://myindex.com", config))
+        self.assertEqual(["https://myindex.com"], config["PLUGIN_INDEXES"])
+
+    def test_add_by_alias(self) -> None:
+        config: Config = {}
+        self.assertTrue(indexes.add("main", config))
+        self.assertEqual(["https://overhang.io/tutor/main"], config["PLUGIN_INDEXES"])
+        self.assertTrue(indexes.remove("main", config))
+        self.assertEqual([], config["PLUGIN_INDEXES"])
+
+    def test_deduplication(self) -> None:
+        plugins = [
+            {"name": "plugin1", "description": "desc1"},
+            {"name": "PLUGIN1", "description": "desc2"},
+        ]
+        deduplicated = indexes.deduplicate_plugins(plugins)
+        self.assertEqual([{"name": "plugin1", "description": "desc2"}], deduplicated)
+
+    def test_short_description(self) -> None:
+        entry = indexes.IndexEntry({"name": "plugin1"})
+        self.assertEqual("", entry.short_description)
+
+    def test_entry_match(self) -> None:
+        self.assertTrue(indexes.IndexEntry({"name": "ecommerce"}).match("ecomm"))
+        self.assertFalse(indexes.IndexEntry({"name": "ecommerce"}).match("ecom1"))
+        self.assertTrue(indexes.IndexEntry({"name": "ecommerce"}).match("Ecom"))
+        self.assertTrue(
+            indexes.IndexEntry(
+                {"name": "ecommerce", "description": "An awesome plugin"}
+            ).match("AWESOME")
+        )

tests/test_utils.py

@@ -232,3 +232,10 @@ class UtilsTests(unittest.TestCase):
             with self.assertRaises(exceptions.TutorError) as e:
                 utils.check_macos_docker_memory()
             self.assertIn("Text encoding error", e.exception.args[0])
+
+    def test_is_http(self) -> None:
+        self.assertTrue(utils.is_http("http://overhang.io/tutor/main"))
+        self.assertTrue(utils.is_http("https://overhang.io/tutor/main"))
+        self.assertFalse(utils.is_http("/home/user/"))
+        self.assertFalse(utils.is_http("home/user/"))
+        self.assertFalse(utils.is_http("http-home/user/"))

tutor/commands/local.py

@@ -94,7 +94,7 @@ Tutor may not work if Docker is configured with < 4 GB RAM. Please follow instru
     if run_upgrade_from_release is not None:
         click.echo(fmt.title("Upgrading from an older release"))
         if not non_interactive:
-            to_release = tutor_env.get_package_release()
+            to_release = tutor_env.get_current_open_edx_release_name()
             question = f"""You are about to upgrade your Open edX platform from {run_upgrade_from_release.capitalize()} to {to_release.capitalize()}
 
 It is strongly recommended to make a backup before upgrading. To do so, run:

tutor/commands/plugins.py

@@ -1,21 +1,24 @@
 from __future__ import annotations
 
 import os
-import urllib.request
+import tempfile
+import typing as t
 
 import click
 import click.shell_completion
 
 from tutor import config as tutor_config
-from tutor import exceptions, fmt, hooks, plugins
+from tutor import exceptions, fmt, hooks, plugins, utils
+from tutor.plugins import indexes
 from tutor.plugins.base import PLUGINS_ROOT, PLUGINS_ROOT_ENV_VAR_NAME
+from tutor.types import Config
 
 from .context import Context
 
 
 class PluginName(click.ParamType):
     """
-    Convenient param type that supports plugin name autocompletion.
+    Convenient param type that supports autocompletion of installed plugin names.
     """
 
     def __init__(self, allow_all: bool = False):
@@ -38,31 +41,94 @@ class PluginName(click.ParamType):
         return [name for name in candidates if name.startswith(incomplete)]
 
 
+class IndexPluginName(click.ParamType):
+    """
+    Param type for auto-completion of plugin names found in index cache.
+    """
+
+    def shell_complete(
+        self, ctx: click.Context, param: click.Parameter, incomplete: str
+    ) -> t.List[click.shell_completion.CompletionItem]:
+        return [
+            click.shell_completion.CompletionItem(entry.name)
+            for entry in indexes.iter_cache_entries()
+            if entry.name.startswith(incomplete.lower())
+        ]
+
+
+class IndexPluginNameOrLocation(IndexPluginName):
+    """
+    Same as IndexPluginName but also auto-completes file location.
+    """
+
+    def shell_complete(
+        self, ctx: click.Context, param: click.Parameter, incomplete: str
+    ) -> t.List[click.shell_completion.CompletionItem]:
+        # Auto-complete plugin names
+        autocompleted = super().shell_complete(ctx, param, incomplete)
+        # Auto-complete local paths
+        autocompleted += click.Path().shell_complete(ctx, param, incomplete)
+        return autocompleted
+
+
 @click.group(
     name="plugins",
     short_help="Manage Tutor plugins",
-    help="Manage Tutor plugins to add new features and customise your Open edX platform",
 )
 def plugins_command() -> None:
     """
-    All plugin commands should work even if there is no existing config file. This is
+    Manage Tutor plugins to add new features and customise your Open edX platform.
-    because users might enable plugins prior to configuration or environment generation.
+
+    Plugins can be downloaded from local and remote indexes. See the `tutor plugins
+    index` subcommand.
+
+    After the plugin index cache has been updated, plugins can be searched with:
+
+        tutor plugins search <pattern>
+
+    Plugins are installed with:
+
+        tutor plugins install <name>
     """
+    # All plugin commands should work even if there is no existing config file. This is
+    # because users might enable or install plugins prior to configuration or
+    # environment generation.
+    # Thus, usage of `config.load_full` is prohibited.
 
 
-@click.command(name="list", help="List installed plugins")
+@click.command(
-def list_command() -> None:
+    short_help="Print the location of file-based plugins",
-    lines = []
+    help=f"""Print the location of yaml-based plugins: nboth python v1 and yaml v0 plugins. This location can be manually
-    first_column_width = 1
+defined by setting the {PLUGINS_ROOT_ENV_VAR_NAME} environment variable""",
+)
+def printroot() -> None:
+    fmt.echo(PLUGINS_ROOT)
+
+
+@click.command(name="list")
+@click.option(
+    "-e",
+    "--enabled",
+    "show_enabled_only",
+    is_flag=True,
+    help="Display enabled plugins only",
+)
+def list_command(show_enabled_only: bool) -> None:
+    """
+    List installed plugins.
+    """
+    plugins_table: list[tuple[str, ...]] = [("NAME", "STATUS", "VERSION")]
     for plugin, plugin_info in plugins.iter_info():
-        plugin_info = plugin_info or ""
+        is_enabled = plugins.is_loaded(plugin)
-        plugin_info.replace("\n", " ")
+        if is_enabled or not show_enabled_only:
-        status = "" if plugins.is_loaded(plugin) else "(disabled)"
+            plugins_table.append(
-        lines.append((plugin, status, plugin_info))
+                (
-        first_column_width = max([first_column_width, len(plugin) + 2])
+                    plugin,
-
+                    plugin_status(plugin),
-    for plugin, status, plugin_info in lines:
+                    (plugin_info or "").replace("\n", " "),
-        print(f"{plugin:{first_column_width}}\t{status:10}\t{plugin_info}")
+                )
+            )
+    fmt.echo(format_table(plugins_table))
 
 
 @click.command(help="Enable a plugin")
@@ -105,48 +171,281 @@ def disable(context: Context, plugin_names: list[str]) -> None:
         )
 
 
-@click.command(
+@click.command(name="update")
-    short_help="Print the location of yaml-based plugins",
+@click.pass_obj
-    help=f"""Print the location of yaml-based plugins. This location can be manually
+def update(context: Context) -> None:
-defined by setting the {PLUGINS_ROOT_ENV_VAR_NAME} environment variable""",
+    """
+    Update the list of available plugins.
+    """
+    config = tutor_config.load(context.root)
+    update_indexes(config)
+
+
+def update_indexes(config: Config) -> None:
+    all_plugins = indexes.fetch(config)
+    cache_path = indexes.save_cache(all_plugins)
+    fmt.echo_info(f"Plugin index local cache: {cache_path}")
+
+
+@click.command()
+@click.argument("names", metavar="name", type=IndexPluginNameOrLocation(), nargs=-1)
+def install(names: list[str]) -> None:
+    """
+    Install one or more plugins.
+
+    Each plugin name can be one of:
+
+    1. A plugin name from the plugin indexes (see `tutor plugins search`)
+    2. A local file that will be copied to the plugins root
+    3. An http(s) location that will be downloaded to the plugins root
+
+    In cases 2. and 3., the plugin root corresponds to the path given by `tutor plugins
+    printroot`.
+    """
+    find_and_install(names, [])
+
+
+@click.command()
+@click.argument("names", metavar="name", type=IndexPluginName(), nargs=-1)
+def upgrade(names: list[str]) -> None:
+    """
+    Upgrade one or more plugins.
+
+    Specify "all" to upgrade all installed plugins. This command will only print a
+    warning for plugins which cannot be found.
+    """
+    if "all" in names:
+        names = list(plugins.iter_installed())
+    available_names = []
+    for name in names:
+        try:
+            indexes.find_in_cache(name)
+        except exceptions.TutorError:
+            fmt.echo_error(
+                f"Failed to upgrade '{name}': plugin could not be found in indexes"
             )
-def printroot() -> None:
-    fmt.echo(PLUGINS_ROOT)
-
-
-@click.command(
-    short_help="Install a plugin",
-    help=f"""Install a plugin, either from a local Python/YAML file or a remote, web-hosted
-location. The plugin will be installed to {PLUGINS_ROOT_ENV_VAR_NAME}.""",
-)
-@click.argument("location", type=click.Path(dir_okay=False))
-def install(location: str) -> None:
-    basename = os.path.basename(location)
-    if not basename.endswith(".yml") and not basename.endswith(".py"):
-        basename += ".py"
-    plugin_path = os.path.join(PLUGINS_ROOT, basename)
-
-    if location.startswith("http"):
-        # Download file
-        response = urllib.request.urlopen(location)
-        content = response.read().decode()
-    elif os.path.isfile(location):
-        # Read file
-        with open(location, encoding="utf-8") as f:
-            content = f.read()
         else:
-        raise exceptions.TutorError(f"No plugin found at {location}")
+            available_names.append(name)
 
+    find_and_install(available_names, ["--upgrade"])
+
+
+def find_and_install(names: list[str], pip_install_opts: t.List[str]) -> None:
+    """
+    Find and install a list of plugins, given by name. Single-file plugins are
+    downloaded/copied. Python packages are or pip-installed.
+    """
+    single_file_plugins = []
+    pip_requirements = []
+    for name in names:
+        if utils.is_url(name):
+            single_file_plugins.append(name)
+        else:
+            plugin = indexes.find_in_cache(name)
+            src = hooks.Filters.PLUGIN_INDEX_ENTRY_TO_INSTALL.apply(plugin.data)[
+                "src"
+            ].strip()
+            if utils.is_url(src):
+                single_file_plugins.append(src)
+            else:
+                # Create requirements file where each plugin reqs is prefixed by a
+                # comment with its name
+                pip_requirements.append(f"# {name}\n{src}")
+
+    for url in single_file_plugins:
+        install_single_file_plugin(url)
+
+    if pip_requirements:
+        # pip install -r reqs.txt
+        requirements_txt = "\n".join(pip_requirements)
+        with tempfile.NamedTemporaryFile(
+            prefix="tutor-reqs-", suffix=".txt", mode="w"
+        ) as tmp_reqs:
+            tmp_reqs.write(requirements_txt)
+            tmp_reqs.flush()
+            fmt.echo_info(f"Installing pip requirements:\n{requirements_txt}")
+            utils.execute(
+                "pip", "install", *pip_install_opts, "--requirement", tmp_reqs.name
+            )
+
+
+def install_single_file_plugin(location: str) -> None:
+    """
+    Download or copy a single file to the plugins root.
+    """
+    plugin_path = os.path.join(PLUGINS_ROOT, os.path.basename(location))
+    if not plugin_path.endswith(".yml") and not plugin_path.endswith(".py"):
+        plugin_path += ".py"
+    # Read url
+    fmt.echo_info(f"Downloading plugin from {location}...")
+    content = utils.read_url(location)
     # Save file
-    if not os.path.exists(PLUGINS_ROOT):
+    utils.ensure_file_directory_exists(plugin_path)
-        os.makedirs(PLUGINS_ROOT)
     with open(plugin_path, "w", newline="\n", encoding="utf-8") as f:
         f.write(content)
     fmt.echo_info(f"Plugin installed at {plugin_path}")
 
 
+@click.command()
+@click.argument("pattern", default="")
+def search(pattern: str) -> None:
+    """
+    Search in plugin descriptions.
+    """
+    results: list[tuple[str, ...]] = [("NAME", "STATUS", "DESCRIPTION")]
+    for plugin in indexes.iter_cache_entries():
+        if plugin.match(pattern):
+            results.append(
+                (
+                    plugin.name,
+                    plugin_status(plugin.name),
+                    plugin.short_description,
+                )
+            )
+    print(format_table(results))
+
+
+@click.command()
+@click.argument("name", type=IndexPluginName())
+def show(name: str) -> None:
+    """
+    Show plugin details from index.
+    """
+    name = name.lower()
+    for plugin in indexes.iter_cache_entries():
+        if plugin.name == name:
+            fmt.echo(
+                f"""Name: {plugin.name}
+Source: {plugin.src}
+Status: {plugin_status(name)}
+Author: {plugin.author}
+Maintainer: {plugin.maintainer}
+Homepage: {plugin.url}
+Index: {plugin.index}
+Description: {plugin.description}"""
+            )
+            return
+    raise exceptions.TutorError(
+        f"No information available for plugin: '{name}'. Plugin could not be found in indexes."
+    )
+
+
+def plugin_status(name: str) -> str:
+    """
+    Return the status of a plugin. Either: "enabled", "installed" or "not installed".
+    """
+    if plugins.is_loaded(name):
+        return "enabled"
+    if plugins.is_installed(name):
+        return "installed"
+    return "not installed"
+
+
+@click.group(name="index", short_help="Manage plugin indexes")
+def index_command() -> None:
+    """
+    Manage plugin indices.
+
+    A plugin index is a list of Tutor plugins. An index can be public and shared with
+    the community, or private, for instance to share plugins with a select group of
+    users. Plugin indexes are a great way to share your plugins with other Tutor users.
+    By default, only the official plugin index is enabled.
+
+    Plugin indexes are fetched by running:
+
+        tutor plugins update
+
+    Plugin index cache is stored locally in the following subdirectory of the Tutor project environment:
+
+        plugins/index/cache.yml
+    """
+
+
+@click.command(name="list", help="List plugin indexes")
+@click.pass_obj
+def index_list(context: Context) -> None:
+    """
+    Print plugin indexes.
+    """
+    config = tutor_config.load(context.root)
+    for index in indexes.get_all(config):
+        fmt.echo(index)
+
+
+@click.command(name="add")
+@click.argument("url", type=click.Path())
+@click.pass_obj
+def index_add(context: Context, url: str) -> None:
+    """
+    Add a plugin index.
+
+    The index URL will be appended with '{version}/plugins.yml'. The index path can be
+    either an http(s) url or a local file path.
+
+    For official indexes, there is no need to pass a full URL. Instead, use "main",
+    "contrib" or "wizard".
+    """
+    config = tutor_config.load_minimal(context.root)
+    if indexes.add(url, config):
+        tutor_config.save_config_file(context.root, config)
+        update_indexes(config)
+    else:
+        fmt.echo_alert("Plugin index was already added")
+
+
+@click.command(name="remove")
+@click.argument("url")
+@click.pass_obj
+def index_remove(context: Context, url: str) -> None:
+    """
+    Remove a plugin index.
+    """
+    config = tutor_config.load_minimal(context.root)
+    if indexes.remove(url, config):
+        tutor_config.save_config_file(context.root, config)
+        update_indexes(config)
+    else:
+        fmt.echo_alert("Plugin index not present")
+
+
+def format_table(rows: t.List[t.Tuple[str, ...]], separator: str = "\t") -> str:
+    """
+    Format a list of values as a tab-separated table. Column sizes are determined such
+    that row values are vertically aligned.
+    """
+    formatted = ""
+    if not rows:
+        return formatted
+    columns_count = len(rows[0])
+    # Determine each column size
+    col_sizes = [1] * columns_count
+    for row in rows:
+        for c, value in enumerate(row):
+            col_sizes[c] = max(col_sizes[c], len(value))
+    # Print all values
+    for r, row in enumerate(rows):
+        for c, value in enumerate(row):
+            if c < len(col_sizes) - 1:
+                formatted += f"{value:{col_sizes[c]}}{separator}"
+            else:
+                # The last column is not left-justified
+                formatted += f"{value}"
+        if r < len(rows) - 1:
+            # Append EOL at all lines but the last one
+            formatted += "\n"
+    return formatted
+
+
+index_command.add_command(index_add)
+index_command.add_command(index_list)
+index_command.add_command(index_remove)
+plugins_command.add_command(index_command)
 plugins_command.add_command(list_command)
+plugins_command.add_command(printroot)
 plugins_command.add_command(enable)
 plugins_command.add_command(disable)
-plugins_command.add_command(printroot)
+plugins_command.add_command(update)
+plugins_command.add_command(search)
 plugins_command.add_command(install)
+plugins_command.add_command(upgrade)
+plugins_command.add_command(show)

tutor/env.py

@@ -370,7 +370,7 @@ def get_env_release(root: str) -> t.Optional[str]:
     return get_release(version)
 
 
-def get_package_release() -> str:
+def get_current_open_edx_release_name() -> str:
     """
     Return the release name associated to this package.
     """

tutor/hooks/catalog.py

@@ -408,6 +408,31 @@ class Filters:
     #: Parameters are the same as for :py:data:`IMAGES_PULL`.
     IMAGES_PUSH: Filter[list[tuple[str, str]], [Config]] = filters.get("images:push")
 
+    #: List of plugin indexes that are loaded when we run `tutor plugins update`. By
+    #: default, the plugin indexes are stored in the user configuration. This filter makes
+    #: it possible to extend and modify this list with plugins.
+    #:
+    #: :parameter list[str] indexes: list of index URLs. Remember that entries further
+    #:   in the list have priority.
+    PLUGIN_INDEXES: Filter[list[str], []] = filters.get("plugins:indexes:entries")
+
+    #: Filter to modify the url of a plugin index url. This is convenient to alias
+    #: plugin indexes with a simple name, such as "main" or "contrib".
+    #:
+    #: :parameter str url: value passed to the `index add/remove` commands.
+    PLUGIN_INDEX_URL: Filter[str, []] = filters.get("plugins:indexes:url")
+
+    #: When installing an entry from a plugin index, the plugin data from the index will
+    #: go through this filter before it is passed along to `pip install`. Thus, this is a
+    #: good place to add custom authentication when you need to install from a private
+    #: index.
+    #:
+    #: :parameter dict[str, str] plugin: the dict entry from the plugin index. It
+    #:   includes an additional "index" key which contains the plugin index URL.
+    PLUGIN_INDEX_ENTRY_TO_INSTALL: Filter[dict[str, str], []] = filters.get(
+        "plugins:indexes:entries:install"
+    )
+
     #: Information about each installed plugin, including its version.
     #: Keep this information to a single line for easier parsing by 3rd-party scripts.
     #:

tutor/plugins/indexes.py

@@ -0,0 +1,245 @@
+from __future__ import annotations
+
+import os
+import typing as t
+
+from yaml.parser import ParserError
+
+from tutor import env, fmt, hooks, serialize, utils
+from tutor.__about__ import __version__, __version_suffix__
+from tutor.exceptions import TutorError
+from tutor.types import Config, get_typed
+
+PLUGIN_INDEXES_KEY = "PLUGIN_INDEXES"
+# Current release name ('zebulon' or 'nightly') and version (1-26)
+RELEASE = __version_suffix__ or env.get_current_open_edx_release_name()
+MAJOR_VERSION = int(__version__.split(".", maxsplit=1)[0])
+
+
+class Indexes:
+    # Store index cache path in this singleton.
+    CACHE_PATH = ""
+
+
+@hooks.Actions.PROJECT_ROOT_READY.add()
+def _set_indexes_cache_path(root: str) -> None:
+    Indexes.CACHE_PATH = env.pathjoin(root, "plugins", "index", "cache.yml")
+
+
+@hooks.Filters.PLUGIN_INDEX_URL.add()
+def _get_index_url_from_alias(url: str) -> str:
+    known_aliases = {
+        "main": "https://overhang.io/tutor/main",
+        "contrib": "https://overhang.io/tutor/contrib",
+        "wizard": "https://overhang.io/tutor/wizard",
+    }
+    return known_aliases.get(url, url)
+
+
+@hooks.Filters.PLUGIN_INDEX_URL.add()
+def _local_absolute_path(url: str) -> str:
+    if os.path.exists(url):
+        url = os.path.abspath(url)
+    return url
+
+
+class IndexEntry:
+    def __init__(self, data: dict[str, str]):
+        self._data = data
+
+    @property
+    def data(self) -> dict[str, str]:
+        return self._data
+
+    @property
+    def name(self) -> str:
+        return self.data["name"]
+
+    @property
+    def src(self) -> str:
+        return self.data["src"]
+
+    @property
+    def short_description(self) -> str:
+        lines = self.description.splitlines() or [""]
+        return lines[0][:128]
+
+    @property
+    def description(self) -> str:
+        return self.data.get("description", "").strip()
+
+    @property
+    def author(self) -> str:
+        return self.data.get("author", "")
+
+    @property
+    def maintainer(self) -> str:
+        return self.data.get("maintainer", "")
+
+    @property
+    def url(self) -> str:
+        return self.data.get("url", "")
+
+    @property
+    def index(self) -> str:
+        return self.data["index"]
+
+    def match(self, pattern: str) -> bool:
+        """
+        Simple case-insensitive pattern matching.
+
+        Pattern matching is case-insensitive. Both the name and description fields are
+        searched.
+        """
+        if not pattern:
+            return True
+        pattern = pattern.lower()
+        if pattern in self.name.lower() or pattern in self.description.lower():
+            return True
+        return False
+
+
+def add(url: str, config: Config) -> bool:
+    """
+    Append an index to the list if not already present.
+
+    Return True if if the list of indexes was modified.
+    """
+    indexes = get_all(config)
+    url = hooks.Filters.PLUGIN_INDEX_URL.apply(url)
+    if url in indexes:
+        return False
+    indexes.append(url)
+    return True
+
+
+def remove(url: str, config: Config) -> bool:
+    """
+    Remove an index to the list if present.
+
+    Return True if if the list of indexes was modified.
+    """
+    indexes = get_all(config)
+    url = hooks.Filters.PLUGIN_INDEX_URL.apply(url)
+    if url not in indexes:
+        return False
+    indexes.remove(url)
+    return True
+
+
+def get_all(config: Config) -> list[str]:
+    """
+    Return the list of all plugin indexes.
+    """
+    config.setdefault(PLUGIN_INDEXES_KEY, [])
+    indexes = get_typed(config, PLUGIN_INDEXES_KEY, list)
+    for url in indexes:
+        if not isinstance(url, str):
+            raise TutorError(
+                f"Invalid plugin index: {url}. Expected 'str', got '{url.__class__}'"
+            )
+    return indexes
+
+
+def fetch(config: Config) -> list[dict[str, str]]:
+    """
+    Fetch the contents of all indexes. Return the list of plugin entries.
+    """
+    all_plugins: list[dict[str, str]] = []
+    indexes = get_all(config)
+    indexes = hooks.Filters.PLUGIN_INDEXES.apply(indexes)
+    for index in indexes:
+        url = named_index_url(index)
+        try:
+            fmt.echo_info(f"Fetching index {url}...")
+            all_plugins += fetch_url(url)
+        except TutorError as e:
+            fmt.echo_error(f"  Failed to update index. {e.args[0]}")
+
+    return deduplicate_plugins(all_plugins)
+
+
+def deduplicate_plugins(plugins: list[dict[str, str]]) -> list[dict[str, str]]:
+    plugins_dict: dict[str, dict[str, str]] = {}
+    for plugin in plugins:
+        # Plugins from later indexes override others
+        plugin["name"] = plugin["name"].lower()
+        plugins_dict[plugin["name"]] = plugin
+
+    return sorted(plugins_dict.values(), key=lambda p: p["name"])
+
+
+def fetch_url(url: str) -> list[dict[str, str]]:
+    content = utils.read_url(url)
+    plugins = parse_index(content)
+    for plugin in plugins:
+        # Store index url in the plugin itself
+        plugin["index"] = url
+    return plugins
+
+
+def parse_index(content: str) -> list[dict[str, str]]:
+    try:
+        plugins = serialize.load(content)
+    except ParserError as e:
+        raise TutorError(f"Could not parse index: {e}") from e
+    validate_index(plugins)
+    valid_plugins = []
+    for plugin in plugins:
+        # check plugin format
+        if "name" not in plugin:
+            fmt.echo_error("  Invalid plugin: missing 'name' attribute")
+        elif not isinstance(plugin["name"], str):
+            fmt.echo_error(
+                f"  Invalid plugin name: expected str, got {plugin['name'].__class__}"
+            )
+        else:
+            valid_plugins.append(plugin)
+    return valid_plugins
+
+
+def validate_index(plugins: t.Any) -> list[dict[str, str]]:
+    if not isinstance(plugins, list):
+        raise TutorError(
+            f"Invalid plugin index format. Expected list, got {plugins.__class__}"
+        )
+    return plugins
+
+
+def named_index_url(url: str) -> str:
+    if utils.is_http(url):
+        separator = "" if url.endswith("/") else "/"
+        return f"{url}{separator}{RELEASE}/plugins.yml"
+    return os.path.join(url, RELEASE, "plugins.yml")
+
+
+def find_in_cache(name: str) -> IndexEntry:
+    """
+    Find entry in cache. If not found, raise error.
+    """
+    name = name.lower()
+    for entry in iter_cache_entries():
+        if entry.name == name:
+            return entry
+    raise TutorError(f"Plugin '{name}' could not be found in indexes")
+
+
+def iter_cache_entries() -> t.Iterator[IndexEntry]:
+    for data in load_cache():
+        yield IndexEntry(data)
+
+
+def save_cache(plugins: list[dict[str, str]]) -> str:
+    env.write_to(serialize.dumps(plugins), Indexes.CACHE_PATH)
+    return Indexes.CACHE_PATH
+
+
+def load_cache() -> list[dict[str, str]]:
+    try:
+        with open(Indexes.CACHE_PATH, encoding="utf8") as cache_if:
+            plugins = serialize.load(cache_if)
+    except FileNotFoundError as e:
+        raise TutorError(
+            f"Local index cache could not be found in {Indexes.CACHE_PATH}. Run `tutor plugins update`."
+        ) from e
+    return validate_index(plugins)

tutor/templates/config/base.yml

@@ -8,3 +8,7 @@ OPENEDX_SECRET_KEY: "{{ 24|random_string }}"
 PLUGINS:
   # The MFE plugin is required
   - mfe
+PLUGIN_INDEXES:
+  # Indexes in this list will be suffixed with the Open edX named version and
+  # "plugins.yml". E.g: https://overhang.io/tutor/main/olive/plugins.yml
+  - https://overhang.io/tutor/main

tutor/types.py

@@ -49,6 +49,6 @@ def get_typed(
     value = config.get(key, default)
     if not isinstance(value, expected_type):
         raise exceptions.TutorError(
-            "Invalid config entry: expected {expected_type.__name__}, got {value.__class__} for key '{key}'"
+            f"Invalid config entry: expected {expected_type.__name__}, got {value.__class__} for key '{key}'"
         )
     return value

tutor/utils.py

@@ -2,6 +2,7 @@ import base64
 import json
 import os
 import random
+import re
 import shlex
 import shutil
 import string
@@ -10,6 +11,8 @@ import subprocess
 import sys
 from functools import lru_cache
 from typing import List, Tuple
+from urllib.error import URLError
+from urllib.request import urlopen
 
 import click
 from Crypto.Protocol.KDF import bcrypt, bcrypt_check
@@ -44,17 +47,23 @@ def ensure_file_directory_exists(path: str) -> None:
     """
     Create file's base directory if it does not exist.
     """
-    directory = os.path.dirname(path)
-    if os.path.isfile(directory):
-        raise exceptions.TutorError(
-            f"Attempting to create a directory, but a file with the same name already exists: {directory}"
-        )
     if os.path.isdir(path):
         raise exceptions.TutorError(
-            f"Attempting to write to a file, but a directory with the same name already exists: {directory}"
+            f"Attempting to write to a file, but a directory with the same name already exists: {path}"
         )
-    if not os.path.exists(directory):
+    ensure_directory_exists(os.path.dirname(path))
-        os.makedirs(directory)
+
+
+def ensure_directory_exists(path: str) -> None:
+    """
+    Create directory if it does not exist.
+    """
+    if os.path.isfile(path):
+        raise exceptions.TutorError(
+            f"Attempting to create a directory, but a file with the same name already exists: {path}"
+        )
+    if not os.path.exists(path):
+        os.makedirs(path)
 
 
 def random_string(length: int) -> str:
@@ -174,7 +183,12 @@ def _docker_compose_command() -> Tuple[str, ...]:
     if shutil.which("docker-compose") is not None:
         return ("docker-compose",)
     if shutil.which("docker") is not None:
-        if subprocess.run(["docker", "compose"], capture_output=True).returncode == 0:
+        if (
+            subprocess.run(
+                ["docker", "compose"], capture_output=True, check=False
+            ).returncode
+            == 0
+        ):
             return ("docker", "compose")
     raise exceptions.TutorError(
         "docker-compose is not installed. Please follow instructions from https://docs.docker.com/compose/install/"
@@ -291,3 +305,43 @@ def check_macos_docker_memory() -> None:
         raise exceptions.TutorError(
             f"Docker is configured to allocate {memory_mib} MiB RAM, less than the recommended {4096} MiB"
         )
+
+
+def read_url(url: str) -> str:
+    """
+    Read an index url, either remote (http/https) or local.
+    """
+    if is_http(url):
+        # web index
+        try:
+            response = urlopen(url)
+            content: str = response.read().decode()
+            return content
+        except URLError as e:
+            raise exceptions.TutorError(f"Request error: {e}") from e
+        except UnicodeDecodeError as e:
+            raise exceptions.TutorError(
+                f"Remote response must be encoded as utf8: {e}"
+            ) from e
+    try:
+        with open(url, encoding="utf8") as f:
+            # local file index
+            return f.read()
+    except FileNotFoundError as e:
+        raise exceptions.TutorError(f"File could not be found: {e}") from e
+    except UnicodeDecodeError as e:
+        raise exceptions.TutorError(f"File must be encoded as utf8: {e}") from e
+
+
+def is_url(text: str) -> bool:
+    """
+    Return true if the string points to a file on disk or a web URL.
+    """
+    return os.path.isfile(text) or is_http(text)
+
+
+def is_http(url: str) -> bool:
+    """
+    Basic test to check whether a string is a web URL. Use only for basic use cases.
+    """
+    return re.match(r"^https?://", url) is not None