Add support for simple, yaml-based plugins

Those plugins are stored as yaml files in ~/.local/share/tutor-plugins
and follow the same specifications as entrypoint plugins.

CHANGELOG.md

@@ -4,11 +4,12 @@ Note: Breaking changes between versions are indicated by "💥".
 
 ## Unreleased
 
-- 💥[Improvement] The output of `plugins list` now includes plugin version numbers.
+- [Feature] Add support for simple, YAML-based plugins
+- 💥[Improvement] The output of `plugins list` now includes plugin version numbers
 
 ## 3.10.1 (2020-01-13)
 
-- [Improvement] Explicitely point to docker.io images, when necessary, for [podman](https://podman.io/) compatibility.
+- [Improvement] Explicitely point to docker.io images, when necessary, for [podman](https://podman.io/) compatibility
 
 ## 3.10.0 (2020-01-10)
 

docs/plugins.rst

@@ -3,15 +3,17 @@
 Plugins
 =======
 
-Since v3.4.0, Tutor comes with a plugin system that allows anyone to customise the deployment of an Open edX platform very easily. The vision behind this plugin system is that users should not have to fork the Tutor repository to customise their deployments. For instance, if you have created a new application that integrates with Open edX, you should not have to describe how to manually patch the platform settings, ``urls.py`` or ``*.env.json`` files. Instead, you can create a "tutor-myapp" plugin for Tutor. Then, users will start using your application in three simple steps::
+Tutor comes with a plugin system that allows anyone to customise the deployment of an Open edX platform very easily. The vision behind this plugin system is that users should not have to fork the Tutor repository to customise their deployments. For instance, if you have created a new application that integrates with Open edX, you should not have to describe how to manually patch the platform settings, ``urls.py`` or ``*.env.json`` files. Instead, you can create a "tutor-myapp" plugin for Tutor. Then, users will start using your application in three simple steps::
 
     # 1) Install the plugin
     pip install tutor-myapp
     # 2) Enable the plugin
     tutor plugins enable myapp
-    # 3) Restart the platform
+    # 3) Reconfigure and restart the platform
     tutor local quickstart
 
+In the following we see how to use and create Tutor plugins.
+
 Commands
 --------
 
@@ -28,30 +30,15 @@ After enabling or disabling a plugin, the environment should be re-generated wit
   
     tutor config save
 
-API (v0)
+API
---------
+---
 
-Note: The API for developing Tutor plugins is still considered unstable: profound changes should be expected for some time.
+Plugins can affect the behaviour of Tutor at multiple levels. First, plugins can define new services with their Docker images, settings and the right initialisation commands. To do so you will have to define custom :ref:`config <plugin_config>`, :ref:`patches <plugin_patches>`, :ref:`hooks <plugin_hooks>` and :ref:`templates <plugin_templates>`. Then, plugins can also extend the CLI by defining their own :ref:`commands <plugin_command>`.
 
-There are two mechanisms by which a plugin can integrate with Tutor: patches and hooks. Patches affect the rendered environment templates, while hooks are actions that are run during the lifetime of an Open edX platform. A plugin indicates which templates it patches, and which hooks it needs to run. A plugin can also affect the project configuration by adding new values and modifying existing values.
+.. _plugin_config:
 
-Entrypoint
+config
-~~~~~~~~~~
+~~~~~~
-
-A plugin is a regular python package with a specific entrypoint: ``tutor.plugin.v0``.
-
-Example::
-  
-    from setuptools import setup
-    setup(
-        ...
-        entry_points={"tutor.plugin.v0": ["myplugin = myplugin.plugin"]},
-    )
-
-The ``myplugin.plugin`` python module should then declare a few attributes that will define its behaviour.
-
-``config``
-~~~~~~~~~~
 
 The ``config`` attribute is used to modify existing and add new configuration parameters:
 
@@ -81,14 +68,20 @@ This configuration from the "myplugin" plugin will set the following values:
 - ``MYPLUGIN_DOCKER_IMAGE``: this value will by default not be stored in ``config.yml``, but ``tutor config printvalue MYPLUGIN_DOCKER_IMAGE`` will print ``username/imagename:latest``.
 - ``MASTER_PASSWORD`` will be set to ``h4cked``. Needless to say, plugin developers should avoid doing this.
 
+.. _plugin_patches:
 
-``patches``
+patches
-~~~~~~~~~~~
+~~~~~~~
 
-The Tutor templates include calls to ``{{ patch("patchname") }}`` in many different places. Plugins can add content in these places by adding values to the ``patches`` attribute.
+Plugin patches affect the rendered environment templates. In many places the Tutor templates include calls to ``{{ patch("patchname") }}``. This grants plugin developers the possibility to modify the content of rendered templates. Plugins can add content in these places by adding values to the ``patches`` attribute.
-
-The ``patches`` attribute can be a callable function instead of a static attribute.
 
+.. note::
+    The list of existing patches can be found by searching for `{{ patch(` strings in the Tutor source code::
+    
+        git grep "{{ patch"
+    
+    The list of patches can also be browsed online `on Github <https://github.com/search?utf8=✓&q={{+patch+repo%3Aoverhangio%2Ftutor+path%3A%2Ftutor%2Ftemplates&type=Code&ref=advsearch&l=&l= 8>`__.
+    
 Example::
   
     patches = {
@@ -98,10 +91,16 @@ Example::
 
 This will add a Redis instance to the services run with ``tutor local`` commands.
 
-``hooks``
+.. note::
-~~~~~~~~~
+    The ``patches`` attribute can be a callable function instead of a static dict value.
 
-Hooks are actions that are run during the lifetime of the platform. Each hook has a different specification.
+
+.. _plugin_hooks:
+
+hooks
+~~~~~
+
+Hooks are actions that are run during the lifetime of the platform. For instance, hooks are used to trigger database initialisation and migrations. Each hook has a different specification.
 
 ``init``
 ++++++++
@@ -155,28 +154,32 @@ Example::
 
 With this hook, users will be able to pull and push the ``myimage:latest`` docker image by running::
       
-        tutor images pull myimage
+    tutor images pull myimage
-        tutor images push myimage
+    tutor images push myimage
 
-    or::
+or::
-      
+  
-        tutor images pull all
+    tutor images pull all
-        tutor images push all
+    tutor images push all
 
-``templates``
+.. _plugin_templates:
-~~~~~~~~~~~~~
+
+templates
+~~~~~~~~~
 
 In order to define plugin-specific hooks, a plugin should also have a template directory that includes the plugin hooks. The ``templates`` attribute should point to that directory.
 
 Example::
   
     import os
-    templates = templates = os.path.join(os.path.abspath(os.path.dirname(__file__)), "templates")
+    templates = os.path.join(os.path.abspath(os.path.dirname(__file__)), "templates")
 
 With the above declaration, you can store plugin-specific templates in the ``templates/myplugin`` folder next to the ``plugin.py`` file.
 
-``command``
+.. _plugin_command:
-~~~~~~~~~~~
+
+command
+~~~~~~~
 
 A plugin can provide custom command line commands. Commands are assumed to be `click.Command <https://click.palletsprojects.com/en/7.x/api/#commands>`__ objects.
 
@@ -212,14 +215,84 @@ This would allow any user to run::
     
 See the official `click documentation <https://click.palletsprojects.com/en/7.x/>`__ for more information.
 
+
 Creating a new plugin
 ---------------------
 
-Creating a new plugin for Tutor is not so hard! You can get started with the `tutor plugin cookiecutter <https://github.com/overhangio/cookiecutter-tutor-plugin>`__::
+Plugins can be created in two different ways: either as plain YAML files or installable Python packages. YAML files are great when you need to make minor changes to the default platform, such as modifying settings. For creating more complex applications, it is recommended to create python packages.
+
+YAML file
+~~~~~~~~~
+
+YAML files that are stored in the tutor plugins root folder will be automatically considered as plugins. The location of the plugin root can be found by running::
+    
+    tutor plugins printroot
+
+On Linux, this points to ``~/.local/share/tutor-plugins``. The location of the plugin root folder can be modified by setting the ``TUTOR_PLUGINS_ROOT`` environment variable.
+
+YAML plugins need to define two extra keys: "name" and "version". Custom CLI commands are not supported by YAML plugins.
+
+Let's create a simple plugin that adds your own `Google Analytics <https://analytics.google.com/>`__ tracking code to your Open edX platform. We need to add the ``GOOGLE_ANALYTICS_ACCOUNT`` and ``GOOGLE_ANALYTICS_TRACKING_ID`` settings to both the LMS and the CMS settings. To do so, we will only have to create the ``openedx-common-settings`` patch, which is shared by the development and the production settings both for the LMS and the CMS. First, create the plugin directory::
+    
+    mkdir "$(tutor plugins printroot)"
+
+Then add the following content to the plugin file located at ``$(tutor plugins printroot)/myplugin.yml``::
+
+    name: myplugin
+    version: 0.1.0
+    patches:
+      openedx-common-settings: |
+        # myplugin special settings
+        GOOGLE_ANALYTICS_ACCOUNT = "UA-654321-1"
+        GOOGLE_ANALYTICS_TRACKING_ID = "UA-654321-1"
+
+Of course, you should replace your Google Analytics tracking code with your own. You can verify that your plugin is correctly installed, but not enabled yet::
+    
+    $ tutor plugins list
+    myplugin@0.1.0 (disabled)
+    
+You can then enable your newly-created plugin::
+    
+    tutor plugins enable myplugin
+
+Update your environment to apply changes from your plugin::
+    
+    tutor config save
+
+You should be able to view your changes in every LMS and CMS settings file::
+
+    grep -r myplugin "$(tutor config printroot)/env/apps/openedx/settings/"
+
+Now just restart your platform to start sending tracking events to Google Analytics::
+    
+    tutor local quickstart
+
+That's it! And it's very easy to share your plugins. Just upload them to your Github repo and share the url with other users. They will be able to install your plugin by running::
+    
+    tutor plugins install https://raw.githubusercontent.com/username/yourrepo/master/myplugin.yml
+
+Python package
+~~~~~~~~~~~~~~
+
+Creating a plugin as a Python package allows you to define more complex logic and to store your patches in a more structured way. Python Tutor plugins are regular Python packages that define a specific entrypoint: ``tutor.plugin.v0``.
+
+Example::
+  
+    from setuptools import setup
+    setup(
+        ...
+        entry_points={"tutor.plugin.v0": ["myplugin = myplugin.plugin"]},
+    )
+
+The ``myplugin.plugin`` python module should then declare the ``config``, ``hooks``, etc. attributes that will define its behaviour.
+
+To get started on the right foot, it is strongly recommended to create your first plugin with the `tutor plugin cookiecutter <https://github.com/overhangio/cookiecutter-tutor-plugin>`__::
 
     pip install cookiecutter
     cookiecutter https://github.com/overhangio/cookiecutter-tutor-plugin.git
-
+    pip install -e ./tutor-myplugin
+    tutor plugins list # your plugin should appear here
+    tutor plugins enable myplugin # hack at it!
 
 .. _existing_plugins:
 

tests/test_plugins.py

@@ -213,3 +213,10 @@ class PluginsTests(unittest.TestCase):
         ):
             plugins2 = plugins.Plugins(config)
             self.assertEqual(1, len(list(plugins2.iter_enabled())))
+
+    def test_dict_plugin(self):
+        plugin = plugins.DictPlugin(
+            {"name": "myplugin", "config": {"set": {"KEY": "value"}}, "version": "0.1"}
+        )
+        self.assertEqual("myplugin", plugin.name)
+        self.assertEqual({"KEY": "value"}, plugin.config_set)

tutor/commands/cli.py

@@ -22,6 +22,18 @@ from .. import fmt
 
 def main():
     try:
+        click_repl.register_repl(cli, name="ui")
+        cli.add_command(images_command)
+        cli.add_command(config_command)
+        cli.add_command(local)
+        cli.add_command(dev)
+        cli.add_command(android)
+        cli.add_command(k8s)
+        cli.add_command(ui)
+        cli.add_command(webui)
+        cli.add_command(print_help)
+        cli.add_command(plugins_command)
+        add_plugin_commands(cli)
         cli()  # pylint: disable=no-value-for-parameter
     except exceptions.TutorError as e:
         fmt.echo_error("Error: {}".format(e.args[0]))
@@ -50,18 +62,5 @@ def print_help():
         click.echo(cli.get_help(context))
 
 
-click_repl.register_repl(cli, name="ui")
-cli.add_command(images_command)
-cli.add_command(config_command)
-cli.add_command(local)
-cli.add_command(dev)
-cli.add_command(android)
-cli.add_command(k8s)
-cli.add_command(ui)
-cli.add_command(webui)
-cli.add_command(print_help)
-cli.add_command(plugins_command)
-add_plugin_commands(cli)
-
 if __name__ == "__main__":
     main()

tutor/commands/plugins.py

@@ -1,9 +1,12 @@
 import os
 import shutil
+import urllib.request
+
 import click
 
 from .. import config as tutor_config
 from .. import env as tutor_env
+from .. import exceptions
 from .. import fmt
 from .. import plugins
 
@@ -66,6 +69,50 @@ def disable(context, plugin_names):
     )
 
 
+@click.command(
+    short_help="Print the location of yaml-based plugins",
+    help="""Print the location of yaml-based plugins. This location can be manually
+defined by setting the {} environment variable""".format(
+        plugins.DictPlugin.ROOT_ENV_VAR_NAME
+    ),
+)
+def printroot():
+    fmt.echo(plugins.DictPlugin.ROOT)
+
+
+@click.command(
+    short_help="Install a plugin",
+    help="""Install a plugin, either from a local YAML file or a remote, web-hosted
+location. The plugin will be installed to {}.""".format(
+        plugins.DictPlugin.ROOT_ENV_VAR_NAME
+    ),
+)
+@click.argument("location")
+def install(location):
+    basename = os.path.basename(location)
+    if not basename.endswith(".yml"):
+        basename += ".yml"
+    plugin_path = os.path.join(plugins.DictPlugin.ROOT, basename)
+
+    if location.startswith("http"):
+        # Download file
+        response = urllib.request.urlopen(location)
+        content = response.read().decode()
+    elif os.path.isfile(location):
+        # Read file
+        with open(location) as f:
+            content = f.read()
+    else:
+        raise exceptions.TutorError("No plugin found at {}".format(location))
+
+    # Save file
+    if not os.path.exists(plugins.DictPlugin.ROOT):
+        os.makedirs(plugins.DictPlugin.ROOT)
+    with open(plugin_path, "w") as f:
+        f.write(content)
+    fmt.echo_info("Plugin installed at {}".format(plugin_path))
+
+
 def add_plugin_commands(command_group):
     """
     Add commands provided by all plugins to the given command group. Each command is
@@ -80,3 +127,5 @@ def add_plugin_commands(command_group):
 plugins_command.add_command(list_command)
 plugins_command.add_command(enable)
 plugins_command.add_command(disable)
+plugins_command.add_command(printroot)
+plugins_command.add_command(install)

tutor/plugins.py

@@ -1,8 +1,14 @@
+from collections import namedtuple
 from copy import deepcopy
+from glob import glob
 import importlib
+import os
 import pkg_resources
 
+import appdirs
+
 from . import exceptions
+from . import serialize
 
 
 CONFIG_KEY = "PLUGINS"
@@ -121,6 +127,39 @@ class OfficialPlugin(BasePlugin):
         yield from cls.INSTALLED
 
 
+class DictPlugin(BasePlugin):
+    ROOT_ENV_VAR_NAME = "TUTOR_PLUGINS_ROOT"
+    ROOT = os.path.expanduser(
+        os.environ.get(ROOT_ENV_VAR_NAME, "")
+    ) or appdirs.user_data_dir(appname="tutor-plugins")
+
+    def __init__(self, data):
+        Module = namedtuple("Module", data.keys())
+        obj = Module(**data)
+        super().__init__(data["name"], obj)
+        self._version = data["version"]
+
+    @property
+    def version(self):
+        return self._version
+
+    @classmethod
+    def iter_installed(cls):
+        for path in glob(os.path.join(cls.ROOT, "*.yml")):
+            with open(path) as f:
+                data = serialize.load(f)
+                if not isinstance(data, dict):
+                    raise exceptions.TutorError(
+                        "Invalid plugin: {}. Expected dict.".format(path)
+                    )
+                try:
+                    yield cls(data)
+                except KeyError as e:
+                    raise exceptions.TutorError(
+                        "Invalid plugin: {}. Missing key: {}".format(path, e.args[0])
+                    )
+
+
 class Plugins:
 
     INSTANCE = None
@@ -161,7 +200,7 @@ class Plugins:
         """
         Iterate on all installed plugins. Plugins are deduplicated by name.
         """
-        classes = [OfficialPlugin, EntrypointPlugin]
+        classes = [OfficialPlugin, EntrypointPlugin, DictPlugin]
         installed_plugin_names = set()
         for PluginClass in classes:
             for plugin in PluginClass.iter_installed():