christianlight
/
tutor
mirror of https://github.com/ChristianLight/tutor.git
7
0
Fork 0
Code Issues Releases Activity

Merge 14612dd094 into 9d3d9e60f5

This commit is contained in:
pull[bot] 2024-04-30 07:22:43 +00:00 committed by GitHub
parent 9d3d9e60f5 14612dd094
commit 68e43c2fb2
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
45 changed files with 451 additions and 144 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

5
.github/workflows/auto-add-to-project.yml vendored
View File

@ -1,9 +1,6 @@
name: Auto Add Issues and Pull Requests to Project
name: Auto Add Issues to Project
on:
pull_request:
types:
- opened
issues:
types:
- opened

7
.github/workflows/test.yml vendored
View File

@ -9,12 +9,15 @@ on:
jobs:
tests:
runs-on: ubuntu-latest
strategy:
matrix:
python-version: ['3.8', '3.12']
steps:
- uses: actions/checkout@v3
- name: Set up Python
uses: actions/setup-python@v3
uses: actions/setup-python@v5
with:
python-version: 3.8
python-version: ${{ matrix.python-version }}
cache: 'pip'
cache-dependency-path: requirements/dev.txt
- name: Upgrade pip

41
CHANGELOG.md
View File

@ -20,6 +20,47 @@ instructions, because git commits are used to generate release notes:
<!-- scriv-insert-here -->
<a id='changelog-17.0.4'></a>
## v17.0.4 (2024-04-09)
- [Security] Update Redis to 7.2.4 (by @dawoudsheraz)
- [Improvement] Update release to open-release/quince.3 (by @dawoudsheraz)
<a id='changelog-17.0.3'></a>
## v17.0.3 (2024-03-26)
- 💥[Bugfix] Prevent infinite growth of course structure cache in Redis. (by @regisb)
- Redis is now configured with a maximum memory size of 4GB. If this is too low for your platform, you should increase this value using the new "redis-conf" patch.
- Make sure that course structure cache keys have an actual timeout.
- [Feature] Introduce the "redis-conf" patch. (by @regisb)
- [Bugfix] Fix merge conflicts in nightly when trying to apply patches from the master branch. (by @regisb)
- [Bugfix] Ensure mounted installable packages are installed as expected upon initialization. (by @dawoudsheraz)
<a id='changelog-17.0.2'></a>
## v17.0.2 (2024-02-09)
- [Feature] Several enhancements to the Demo Course (by @kdmccormick):
- The [Open edX Demo Course](https://github.com/openedx/openedx-demo-course) has been re-built from scratch with up-to-date instruction-focused content. Its directory structure has changed.
- In order to support both the old and new structures of the Demo Course's repository, the command `tutor local do importdemocourse` will now auto-determine the course root based on the location of `course.xml`. Use the `--repo-dir` argument to override this behavior.
- The new command `tutor local do importdemolibraries` will import any content libraries defined within the Demo Course repository. At the moment, that is just the "Respiratory System Question Bank", which is an optional but helpful extension to the new Demo Course.
- To try out the new Demo Course now, run: `tutor local do importdemocourse --version master`.
- To try out the demo Respiratory System Question Bank now, run: `tutor local do importdemolibraries --version master`.
- To revert back to an older Demo Course version at any point, run: `tutor local do importdemocourse --version open-release/quince.2`, replacing `quince.2` with your preferred course version.
- [Bugfix] Remove duplicate volume declarations that cause `docker compose` v2.24.1 to fail.
- [Bugfix] Actually update the environment on `tutor plugins enable ...`. (by @regisb)
- [Feature] Introduce a `tutor.hooks.lru_cache` decorator that is automatically cleared whenever a plugin is loaded or unloaded. This is useful, in particular when a plugin implements a costly function that depends on tutor hooks. (by @regisb)
- [Bugfix] Fix compatibility with Python 3.12 by replacing pkg_resources with importlib_metadata and importlib_resources. (by @Danyal-Faheem)
- [Improvement] Upgrade base release to open-release/quince.2. (by @regisb)
<a id='changelog-17.0.1'></a>
## v17.0.1 (2024-01-25)
- [Bugfix] Error "'Crypto.PublicKey.RSA.RsaKey object' has no attribute 'dq'" during `tutor config save` was caused by outdated minimum version of the pycryptodome package. To resolve this issue, run `pip install --upgrade pycryptodome`. (by @regisb)
- [Feature] add `CONFIG_INTERACTIVE` action that allows tutor plugins to interact with the configuration at the time of the interactive questionnaire that happens during tutor local launch. (by @Alec4r).
- [Improvement] Add `.webp` and. `.otf` extensions to list of binary extensions to ignore when rendering templates.
- [Security] Fix JWT scopes in XBlock callbacks. (by @regisb)
<a id='changelog-17.0.0'></a>
## v17.0.0 (2023-12-09)

19
SECURITY.md Normal file
View File

@ -0,0 +1,19 @@
# Tutor Ethical Vulnerability Disclosure Policy
## Reporting a Vulnerability
To ensure the health of the codebase and the larger Open edX and Tutor communities, please do not create GitHub issues for a security vulnerability. Report any security vulnerabilities or concerns by sending an email to [security.tutor@edly.io](mailto:security.tutor@edly.io). To ensure a timely triage and fix of the security issue, include as many details you can when reporting the vulnerability. Some pieces of information to consider:
* The nature of the vulnerability, e.g.
* Authentication and Authorization
* Data Integrity and Confidentiality
* Security Configurations
* Third-party dependencies
* The impact of the security risk
* A detailed description of the steps necessary to reproduce the issue
* The links to the vulnerable code
* The links to third-party libraries/packages if the vulnerability is present in such a dependency.
## Bug Bounty
Edly/Tutor does not offer a bug bounty for reported vulnerabilities.

1
changelog.d/20231214_110904_regis_rsa_error.md
View File

@ -1 +0,0 @@
- [Bugfix] Error "'Crypto.PublicKey.RSA.RsaKey object' has no attribute 'dq'" during `tutor config save` was caused by outdated minimum version of the pycryptodome package. To resolve this issue, run `pip install --upgrade pycryptodome`. (by @regisb)

1
changelog.d/20231214_141533_alecar.main_survey_report_plugin.md
View File

@ -1 +0,0 @@
- [Feature] add `CONFIG_INTERACTIVE` action that allows tutor plugins to interact with the configuration at the time of the interactive questionnaire that happens during tutor local launch. (by @Alec4r).

1
changelog.d/20240403_155328_r.khetani_improvement_patch_proxy_directive.md Normal file
View File

@ -0,0 +1 @@
[Improvement] Add ability to patch proxy configuration for Caddy (by @ravikhetani)

11
docs/conf.py
View File

@ -51,6 +51,8 @@ nitpick_ignore = [
# python 3.10
("py:class", "NoneType"),
("py:class", "click.core.Command"),
# Python 3.12
("py:class", "FilterCallbackFunc"),
]
# Resolve type aliases here
# https://www.sphinx-doc.org/en/master/usage/extensions/autodoc.html#confval-autodoc_type_aliases
@ -58,6 +60,15 @@ autodoc_type_aliases: dict[str, str] = {
# python 3.10
"T": "tutor.core.hooks.actions.T",
"T2": "tutor.core.hooks.filters.T2",
# # python 3.12
"L": "tutor.core.hooks.filters.L",
"FilterCallbackFunc": "tutor.core.hooks.filters.FilterCallbackFunc",
# https://stackoverflow.com/questions/73223417/type-aliases-in-type-hints-are-not-preserved
# https://github.com/sphinx-doc/sphinx/issues/10455
# https://github.com/sphinx-doc/sphinx/issues/10785
# https://github.com/emdgroup/baybe/pull/67
"Action": "tutor.core.hooks.actions.Action",
"Filter": "tutor.core.hooks.filters.Filter",
}

2
docs/configuration.rst
View File

@ -87,7 +87,7 @@ This configuration parameter defines which MySQL Docker image to use.
.. https://hub.docker.com/_/redis/tags
- ``DOCKER_IMAGE_REDIS`` (default: ``"docker.io/redis:7.0.11"``)
- ``DOCKER_IMAGE_REDIS`` (default: ``"docker.io/redis:7.2.4"``)
This configuration parameter defines which Redis Docker image to use.

22
docs/plugins/examples.rst
View File

@ -43,18 +43,18 @@ Enable Google Analytics
::
from tutor import hooks
from tutor import hooks
hooks.Filters.ENV_PATCHES.add_items([
(
"openedx-common-settings",
"GOOGLE_ANALYTICS_4_ID = 'MY-MEASUREMENT-ID'"
),
(
"mfe-lms-common-settings",
"MFE_CONFIG['GOOGLE_ANALYTICS_4_ID'] = 'MY-MEASUREMENT-ID'"
),
])
hooks.Filters.ENV_PATCHES.add_items([
(
"openedx-common-settings",
"GOOGLE_ANALYTICS_4_ID = 'MY-MEASUREMENT-ID'"
),
(
"mfe-lms-common-settings",
"MFE_CONFIG['GOOGLE_ANALYTICS_4_ID'] = 'MY-MEASUREMENT-ID'"
),
])
.. note::
Please be aware that as of May 2023 Google Analytics support has been upgraded from Google Universal Analytics to Google Analytics 4 and you may need to update your configuration as mentioned in the `Open edX docs <https://docs.openedx.org/en/latest/site_ops/how-tos/google-analytics.html>`__.

6
docs/reference/api/hooks/catalog.rst
View File

@ -16,9 +16,3 @@ The underlying Python hook classes and API are documented :ref:`here <hooks_api>
.. autoclass:: tutor.hooks.Contexts
:members:
Open edX hooks
--------------
.. automodule:: tutor.plugins.openedx.hooks
:members:

24
docs/reference/api/hooks/index.rst
View File

@ -1,8 +1,11 @@
.. _hooks_api:
==========
Hook types
==========
=========
Hooks API
=========
Types
=====
This is the Python documentation of the two types of hooks (actions and filters) as well as the contexts system which is used to instrument them. Understanding how Tutor hooks work is useful to create plugins that modify the behaviour of Tutor. However, plugin developers should almost certainly not import these hook types directly. Instead, use the reference :ref:`hooks catalog <hooks_catalog>`.
@ -12,3 +15,18 @@ This is the Python documentation of the two types of hooks (actions and filters)
actions
filters
contexts
Utilities
=========
Functions
---------
.. autofunction:: tutor.core.hooks::clear_all
.. autofunction:: tutor.hooks::lru_cache
Priorities
----------
.. automodule:: tutor.core.hooks.priorities
:members: HIGH, DEFAULT, LOW

14
docs/reference/patches.rst
View File

@ -52,6 +52,13 @@ File: ``apps/caddy/Caddyfile``
File: ``apps/caddy/Caddyfile``
.. patch:: caddyfile-proxy
``caddyfile-proxy``
===========================
File: ``apps/caddy/Caddyfile``
.. patch:: cms-env
``cms-env``
@ -376,6 +383,13 @@ File: ``apps/openedx/settings/lms/production.py``
Python-formatted LMS settings in production. Values defined here override the values from :patch:`openedx-lms-common-settings`.
``redis-conf``
==============
File: ``apps/redis/redis.conf``
Implement this patch to override hard-coded Redis configuration values. See the `Redis configuration reference <https://redis.io/docs/management/config-file/>`__`.
``uwsgi-config``
================

14
docs/troubleshooting.rst
View File

@ -169,6 +169,8 @@ This issue should only happen in development mode. Long story short, it can be s
If you'd like to learn more, please take a look at `this Github issue <https://github.com/overhangio/tutor/issues/302>`__.
.. _high_resource_consumption:
High resource consumption on ``tutor images build`` by docker
-------------------------------------------------------------
@ -193,3 +195,15 @@ Now build again::
tutor images build
All build commands should now make use of the newly configured builder. To later revert to the default builder, run ``docker buildx use default``.
.. note::
Setting a too low value for maximum parallelism will result in longer build times.
fatal: the remote end hung up unexpectedly / fatal: early EOF / fatal: index-pack failed when running ``tutor images build ...``
--------------------------------------------------------------------------------------------------------------------------------
This issue can occur due to problems with the network connection while cloning edx-platform which is a fairly large repository.
First, try to run the same command once again to see if it works as the network connection can sometimes drop during the build process.
If that does not work, follow the tutorial above for :ref:`High resource consumption <high_resource_consumption>` to limit the number of concurrent build steps so that the network connection is not being shared between multiple layers at once.

15
docs/tutor.rst
View File

@ -68,9 +68,18 @@ You can then browse the documentation with::
Releasing a new version
~~~~~~~~~~~~~~~~~~~~~~~
- Bump the ``__version__`` value in ``tutor/__about__.py``. (see :ref:`versioning` below)
- Collect changelog entries with ``make changelog``.
- Create a commit with the version changelog.
Releasing a version includes two phases:
1. Add changes and generate individual changelog entries:
- run ``make changelog-entry``(or ``scriv create``) command - It will create changelog entries in a folder named changelog.d for the changes user has done for releasing a new version.
- Commit and merge all the changes including the changelog entries. e.g this `commit <https://github.com/overhangio/tutor-discovery/commit/e30a78936d63439bde069aeff11960585bd81592>`__.
2. Update version and compile changelogs:
- Now bump the ``__version__`` value in ``tutor/__about__.py``. (see :ref:`versioning` below).
- Collect changelog entries with ``make changelog``(or ``scriv collect``) command - It will delete all previous changelog entries from changelog.d folder and will add records of those entries to CHANGELOG.md file.
- Create a commit with the version changelog e.g. this `commit <https://github.com/overhangio/tutor-discovery/commit/18cce706a794c4968e713f0f72c6b912a2ff1e53>`__.
- Run tests with ``make test``.
- Push your changes to the upstream repository.

2
docs/tutorials/edx-platform.rst
View File

@ -160,7 +160,7 @@ It is quite possible that your package is not automatically recognized and bind-
To do so, you will need to create a :ref:`Tutor plugin <plugin_development_tutorial>` that implements the :py:data:`tutor.hooks.Filters.MOUNTED_DIRECTORIES` filter::
import tutor import hooks
from tutor import hooks
hooks.Filters.MOUNTED_DIRECTORIES.add_item(("openedx", "my-package"))
After you implement and enable that plugin, ``tutor mounts list`` should display your directory among the bind-mounted directories.

13
docs/tutorials/google-smtp.rst
View File

@ -6,7 +6,18 @@ By default, Tutor comes with a simple SMTP server for sending emails. Such a ser
.. warning::
Google Mail SMTP servers come with their own set of limitations. For instance, you are limited to sending 500 emails a day. Reference: https://support.google.com/mail/answer/22839
You should authorize third-party to access your Google Mail account. In your Google Mail account, select "Manage Account", "Security", and turn on "Less Secure App Access". Check the Google documentation for more information on "less secure apps": https://support.google.com/accounts/answer/6010255
Authorization for Third-Party Access :
To enhance security, Google recommends the use of "Application-Specific Passwords" for third-party access to Google services. It's crucial to follow these steps to enable this feature:
1. Activate 2-Step Verification for the Google Account. This is essential for setting up application-specific passwords.
2. Visit the Google Account Security page.
3. Under 'Signing in to Google,' select 'App passwords.'
4. It may be necessary to sign in again. After signing in, choose "Select app" and select "Other (Custom name)" from the dropdown menu.
5. Enter a name that describes the purpose of this password, such as 'Tutor SMTP'.
6. Click 'Generate' to receive your 16-character app-specific password. Make sure to record this password securely.
Reference: https://support.google.com/mail/answer/185833
Then, check that you can reach the Google Mail SMTP service from your own server::

2
requirements/base.in
View File

@ -6,3 +6,5 @@ mypy
pycryptodome>=3.17.0
pyyaml>=6.0
typing-extensions>=4.4.0
importlib-metadata>=7.0.1
importlib-resources>=6.1.1

12
requirements/base.txt
View File

@ -20,7 +20,11 @@ google-auth==2.23.3
# via kubernetes
idna==3.4
# via requests
jinja2==3.1.2
importlib-metadata==7.0.1
# via -r requirements/base.in
importlib-resources==6.1.1
# via -r requirements/base.in
jinja2==3.1.3
# via -r requirements/base.in
kubernetes==28.1.0
# via -r requirements/base.in
@ -40,7 +44,7 @@ pyasn1==0.5.0
# rsa
pyasn1-modules==0.3.0
# via google-auth
pycryptodome==3.19.0
pycryptodome==3.20.0
# via -r requirements/base.in
python-dateutil==2.8.2
# via kubernetes
@ -72,3 +76,7 @@ urllib3==1.26.18
# requests
websocket-client==1.6.4
# via kubernetes
zipp==3.17.0
# via
# importlib-metadata
# importlib-resources

14
requirements/dev.txt
View File

@ -42,7 +42,7 @@ click-log==0.4.0
# via scriv
coverage==7.3.2
# via -r requirements/dev.in
cryptography==41.0.7
cryptography==42.0.3
# via secretstorage
dill==0.3.7
# via pylint
@ -58,14 +58,17 @@ idna==3.4
# via
# -r requirements/base.txt
# requests
importlib-metadata==6.8.0
importlib-metadata==7.0.1
# via
# -r requirements/base.txt
# build
# keyring
# pyinstaller
# twine
importlib-resources==6.1.1
# via keyring
# via
# -r requirements/base.txt
# keyring
isort==5.12.0
# via pylint
jaraco-classes==3.3.0
@ -74,7 +77,7 @@ jeepney==0.8.0
# via
# keyring
# secretstorage
jinja2==3.1.2
jinja2==3.1.3
# via
# -r requirements/base.txt
# scriv
@ -136,7 +139,7 @@ pyasn1-modules==0.3.0
# google-auth
pycparser==2.21
# via cffi
pycryptodome==3.19.0
pycryptodome==3.20.0
# via -r requirements/base.txt
pygments==2.16.1
# via
@ -232,6 +235,7 @@ wheel==0.41.2
# via pip-tools
zipp==3.17.0
# via
# -r requirements/base.txt
# importlib-metadata
# importlib-resources

17
requirements/docs.txt
View File

@ -42,9 +42,13 @@ idna==3.4
# requests
imagesize==1.4.1
# via sphinx
importlib-metadata==6.8.0
# via sphinx
jinja2==3.1.2
importlib-metadata==7.0.1
# via
# -r requirements/base.txt
# sphinx
importlib-resources==6.1.1
# via -r requirements/base.txt
jinja2==3.1.3
# via
# -r requirements/base.txt
# sphinx
@ -76,7 +80,7 @@ pyasn1-modules==0.3.0
# via
# -r requirements/base.txt
# google-auth
pycryptodome==3.19.0
pycryptodome==3.20.0
# via -r requirements/base.txt
pygments==2.16.1
# via sphinx
@ -153,4 +157,7 @@ websocket-client==1.6.4
# -r requirements/base.txt
# kubernetes
zipp==3.17.0
# via importlib-metadata
# via
# -r requirements/base.txt
# importlib-metadata
# importlib-resources

1
requirements/plugins.txt
View File

@ -5,6 +5,7 @@ tutor-credentials>=17.0.0,<18.0.0
tutor-discovery>=17.0.0,<18.0.0
tutor-ecommerce>=17.0.0,<18.0.0
tutor-forum>=17.0.0,<18.0.0
tutor-indigo>=17.0.0,<18.0.0
tutor-jupyter>=17.0.0,<18.0.0
tutor-mfe>=17.0.0,<18.0.0
tutor-minio>=17.0.0,<18.0.0

27
tests/commands/test_jobs.py
View File

@ -38,7 +38,32 @@ class JobsTests(PluginsTestCase, TestCommandMixin):
self.assertEqual(0, result.exit_code)
self.assertIn("cms-job", dc_args)
self.assertIn(
"git clone https://github.com/openedx/edx-demo-course", dc_args[-1]
"git clone https://github.com/openedx/openedx-demo-course", dc_args[-1]
)
def test_import_demo_libraries(self) -> None:
with temporary_root() as root:
self.invoke_in_root(root, ["config", "save"])
with patch("tutor.utils.docker_compose") as mock_docker_compose:
result = self.invoke_in_root(
root,
[
"local",
"do",
"importdemolibraries",
"admin",
],
)
dc_args, _dc_kwargs = mock_docker_compose.call_args
self.assertIsNone(result.exception)
self.assertEqual(0, result.exit_code)
self.assertIn("cms-job", dc_args)
self.assertIn(
"git clone https://github.com/openedx/openedx-demo-course", dc_args[-1]
)
self.assertIn(
"./manage.py cms import_content_library /tmp/library.tar.gz admin",
dc_args[-1],
)
def test_set_theme(self) -> None:

1
tests/openedx-lms-common-settings
View File

@ -1 +0,0 @@
ORA2_FILEUPLOAD_BACKEND = "s3"

16
tests/test_plugins.py Normal file
View File

@ -0,0 +1,16 @@
from __future__ import annotations
from tests.helpers import PluginsTestCase
from tutor import hooks, plugins
class PluginsTests(PluginsTestCase):
def test_env_patches_updated_on_new_plugin(self) -> None:
self.assertEqual([], list(plugins.iter_patches("mypatch")))
hooks.Filters.ENV_PATCHES.add_item(("mypatch", "hello!"))
# env patches cache should be cleared on new plugin
hooks.Actions.PLUGIN_LOADED.do("dummyplugin")
self.assertEqual(["hello!"], list(plugins.iter_patches("mypatch")))

2
tests/test_plugins_v0.py
View File

@ -9,7 +9,7 @@ from tutor.plugins import v0 as plugins_v0
from tutor.types import Config, get_typed
class PluginsTests(PluginsTestCase):
class PluginsV0Tests(PluginsTestCase):
def test_iter_installed(self) -> None:
self.assertEqual([], list(plugins.iter_installed()))

8
tutor.spec
View File

@ -1,7 +1,7 @@
# -*- mode: python -*-
import importlib
import os
import pkg_resources
import importlib_metadata
block_cipher = None
@ -10,16 +10,16 @@ hidden_imports = []
# Auto-discover plugins and include patches & templates folders
for entrypoint_version in ["tutor.plugin.v0", "tutor.plugin.v1"]:
for entrypoint in pkg_resources.iter_entry_points(entrypoint_version):
for entrypoint in importlib_metadata.entry_points(group=entrypoint_version):
plugin_name = entrypoint.name
try:
plugin = entrypoint.load()
plugin = importlib.import_module(entrypoint.value)
except Exception as e:
print(f"ERROR Failed to load plugin {plugin_name}: {e}")
continue
plugin_root = os.path.dirname(plugin.__file__)
plugin_root_module_name = os.path.basename(plugin_root)
hidden_imports.append(entrypoint.module_name)
hidden_imports.append(entrypoint.module)
for folder in ["patches", "templates"]:
path = os.path.join(plugin_root, folder)
if os.path.exists(path):

2
tutor/__about__.py
View File

@ -2,7 +2,7 @@ import os
# Increment this version number to trigger a new release. See
# docs/tutor.html#versioning for information on the versioning scheme.
__version__ = "17.0.0"
__version__ = "17.0.4"
# The version suffix will be appended to the actual version, separated by a
# dash. Use this suffix to differentiate between the actual released version and

78
tutor/commands/jobs.py
View File

@ -45,7 +45,7 @@ def _add_core_init_tasks() -> None:
hooks.Filters.CLI_DO_INIT_TASKS.add_item(
(
"lms",
env.read_core_template_file("jobs", "init", "mounted-edx-platform.sh"),
env.read_core_template_file("jobs", "init", "mounted-directories.sh"),
),
# If edx-platform is mounted, then we may need to perform some setup
# before other initialization scripts can be run.
@ -124,7 +124,7 @@ u.save()"
@click.option(
"-r",
"--repo",
default="https://github.com/openedx/edx-demo-course",
default="https://github.com/openedx/openedx-demo-course",
show_default=True,
help="Git repository that contains the course to be imported",
)
@ -133,7 +133,7 @@ u.save()"
"--repo-dir",
default="",
show_default=True,
help="Git relative subdirectory to import data from",
help="Git relative subdirectory to import data from. If unspecified, will default to the directory containing course.xml",
)
@click.option(
"-v",
@ -145,15 +145,82 @@ def importdemocourse(
) -> t.Iterable[tuple[str, str]]:
version = version or "{{ OPENEDX_COMMON_VERSION }}"
template = f"""
# Import demo course
# Clone the repo
git clone {repo} --branch {version} --depth 1 /tmp/course
python ./manage.py cms import ../data /tmp/course/{repo_dir}
# Determine root directory for course import. If one is provided, use that.
# Otherwise, use the directory containing course.xml, failing if there isn't exactly one.
if [ -n "{repo_dir}" ] ; then
course_root=/tmp/course/{repo_dir}
else
course_xml_first="$(find /tmp/course -name course.xml | head -n 1)"
course_xml_extra="$(find /tmp/course -name course.xml | tail -n +2)"
echo "INFO: Found course.xml files(s): $course_xml_first $course_xml_extra"
if [ -z "$course_xml_first" ] ; then
echo "ERROR: Could not find course.xml. Are you sure this is the right repository?"
exit 1
fi
if [ -n "$course_xml_extra" ] ; then
echo "ERROR: Found multiple course.xml files--course root is ambiguous!"
echo " Please specify a course root dir (relative to repo root) using --repo-dir."
exit 1
fi
course_root="$(dirname "$course_xml_first")"
fi
echo "INFO: Will import course data at: $course_root" && echo
# Import into CMS
python ./manage.py cms import ../data "$course_root"
# Re-index courses
./manage.py cms reindex_course --all --setup"""
yield ("cms", template)
@click.command(help="Import the demo content libraries")
@click.argument("owner_username")
@click.option(
"-r",
"--repo",
default="https://github.com/openedx/openedx-demo-course",
show_default=True,
help="Git repository that contains the library/libraries to be imported",
)
@click.option(
"-v",
"--version",
help="Git branch, tag or sha1 identifier. If unspecified, will default to the value of the OPENEDX_COMMON_VERSION setting.",
)
def importdemolibraries(
owner_username: str, repo: str, version: t.Optional[str]
) -> t.Iterable[tuple[str, str]]:
version = version or "{{ OPENEDX_COMMON_VERSION }}"
template = f"""
# Clone the repo
git clone {repo} --branch {version} --depth 1 /tmp/library
# Fail loudly if:
# * there no library.xml files, or
# * any library.xml is not within a directory named "library/" (upstream edx-platform expectation).
if ! find /tmp/library -name library.xml | grep -q "." ; then
echo "ERROR: No library.xml files found in repository. Are you sure this is the right repository and version?"
exit 1
fi
# For every library.xml file, create a tar of its parent directory, and import into CMS.
for lib_root in $(find /tmp/library -name library.xml | xargs dirname) ; do
echo "INFO: Will import library at $lib_root"
if [ "$(basename "$lib_root")" != "library" ] ; then
echo "ERROR: can only import library.xml files that are within a directory named 'library'"
exit 1
fi
rm -rf /tmp/library.tar.gz
( cd "$(dirname "$lib_root")" && tar czvf /tmp/library.tar.gz library )
yes | ./manage.py cms import_content_library /tmp/library.tar.gz {owner_username}
done"""
yield ("cms", template)
@click.command(
name="print-edx-platform-setting",
help="Print the value of an edx-platform Django setting.",
@ -324,6 +391,7 @@ hooks.Filters.CLI_DO_COMMANDS.add_items(
[
createuser,
importdemocourse,
importdemolibraries,
initialise,
print_edx_platform_setting,
settheme,

1
tutor/commands/plugins.py
View File

@ -1,7 +1,6 @@
from __future__ import annotations
import os
import sys
import tempfile
import typing as t

4
tutor/core/hooks/priorities.py
View File

@ -4,8 +4,12 @@ import typing as t
from typing_extensions import Protocol
#: High priority callbacks are triggered first.
HIGH = 5
#: By default, all callbacks have the same priority and are processed in the order they
#: were added.
DEFAULT = 10
#: Low-priority callbacks are called last. Add callbacks with this priority to override previous callbacks. To add callbacks with even lower priority, use ``LOW + somevalue`` (though such behaviour is not encouraged).
LOW = 50

16
tutor/env.py
View File

@ -7,15 +7,25 @@ import typing as t
from copy import deepcopy
import jinja2
import pkg_resources
import importlib_resources
from tutor import exceptions, fmt, hooks, plugins, utils
from tutor.__about__ import __app__, __version__
from tutor.types import Config, ConfigValue
TEMPLATES_ROOT = pkg_resources.resource_filename("tutor", "templates")
TEMPLATES_ROOT = str(importlib_resources.files("tutor") / "templates")
VERSION_FILENAME = "version"
BIN_FILE_EXTENSIONS = [".ico", ".jpg", ".patch", ".png", ".ttf", ".woff", ".woff2"]
BIN_FILE_EXTENSIONS = [
".ico",
".jpg",
".otf",
".patch",
".png",
".ttf",
".webp",
".woff",
".woff2",
]
JinjaFilter = t.Callable[..., t.Any]

27
tutor/hooks/__init__.py
View File

@ -2,8 +2,35 @@
__license__ = "Apache 2.0"
import typing as t
import functools
from typing_extensions import ParamSpec
# The imports that follow are the hooks API
from tutor.core.hooks import clear_all, priorities
from tutor.types import Config
from .catalog import Actions, Contexts, Filters
def lru_cache(func: t.Callable[..., t.Any]) -> t.Callable[..., t.Any]:
"""
LRU cache decorator similar to `functools.lru_cache
<https://docs.python.org/3/library/functools.html#functools.lru_cache>`__ that is
automatically cleared whenever plugins are updated.
Use this to decorate functions that need to be called multiple times with a return
value that depends on which plugins are loaded. Typically: functions that depend on
the output of filters.
"""
decorated = functools.lru_cache(func)
@Actions.PLUGIN_LOADED.add()
def _clear_func_cache_on_load(_plugin: str) -> None:
decorated.cache_clear()
@Actions.PLUGIN_UNLOADED.add()
def _clear_func_cache_on_unload(_plugin: str, _root: str, _config: Config) -> None:
decorated.cache_clear()
return decorated

47
tutor/plugins/__init__.py
View File

@ -3,6 +3,7 @@ Provide API for plugin features.
"""
from __future__ import annotations
import functools
import typing as t
from copy import deepcopy
@ -12,38 +13,6 @@ from tutor.types import Config, get_typed
# Import modules to trigger hook creation
from . import openedx, v0, v1
# Cache of plugin patches, for efficiency
ENV_PATCHES_DICT: dict[str, list[str]] = {}
@hooks.Actions.PLUGINS_LOADED.add()
def _fill_patch_cache_on_load() -> None:
"""
This action is run after plugins have been loaded.
"""
_fill_patches_cache()
@hooks.Actions.PLUGIN_UNLOADED.add()
def _fill_patch_cache_on_unload(plugin: str, root: str, _config: Config) -> None:
"""
This action is run after plugins have been unloaded.
"""
_fill_patches_cache()
def _fill_patches_cache() -> None:
"""
Some patches are added as (name, content) tuples with the ENV_PATCHES
filter. We convert these patches to add them to ENV_PATCHES_DICT. This makes it
easier for end-user to declare patches, and it's more performant.
"""
ENV_PATCHES_DICT.clear()
patches: t.Iterable[tuple[str, str]] = hooks.Filters.ENV_PATCHES.iterate()
for name, content in patches:
ENV_PATCHES_DICT.setdefault(name, [])
ENV_PATCHES_DICT[name].append(content)
def is_installed(name: str) -> bool:
"""
@ -127,7 +96,19 @@ def iter_patches(name: str) -> t.Iterator[str]:
"""
Yields: patch (str)
"""
yield from ENV_PATCHES_DICT.get(name, [])
yield from _env_patches().get(name, [])
@hooks.lru_cache
def _env_patches() -> dict[str, list[str]]:
"""
Dictionary of patches, implemented for performance reasons.
"""
patches: dict[str, list[str]] = {}
for name, content in hooks.Filters.ENV_PATCHES.iterate():
patches.setdefault(name, [])
patches[name].append(content)
return patches
def unload(plugin: str) -> None:

3
tutor/plugins/openedx.py
View File

@ -4,8 +4,7 @@ import os
import re
import typing as t
from tutor import bindmount
from tutor import hooks
from tutor import bindmount, hooks
from tutor.__about__ import __version_suffix__

13
tutor/plugins/v0.py
View File

@ -5,7 +5,7 @@ import typing as t
from glob import glob
import click
import pkg_resources
import importlib_metadata
from tutor import env, exceptions, fmt, hooks, serialize
from tutor.__about__ import __app__
@ -246,12 +246,12 @@ class EntrypointPlugin(BasePlugin):
ENTRYPOINT = "tutor.plugin.v0"
def __init__(self, entrypoint: pkg_resources.EntryPoint) -> None:
self.loader: pkg_resources.EntryPoint
def __init__(self, entrypoint: importlib_metadata.EntryPoint) -> None:
self.loader: importlib_metadata.EntryPoint = entrypoint
super().__init__(entrypoint.name, entrypoint)
def _load_obj(self) -> None:
self.obj = self.loader.load()
self.obj = importlib.import_module(self.loader.value)
def _version(self) -> t.Optional[str]:
if not self.loader.dist:
@ -260,12 +260,11 @@ class EntrypointPlugin(BasePlugin):
@classmethod
def discover_all(cls) -> None:
for entrypoint in pkg_resources.iter_entry_points(cls.ENTRYPOINT):
entrypoints = importlib_metadata.entry_points(group=cls.ENTRYPOINT)
for entrypoint in entrypoints:
try:
error: t.Optional[str] = None
cls(entrypoint)
except pkg_resources.VersionConflict as e:
error = e.report()
except Exception as e: # pylint: disable=broad-except
error = str(e)
if error:

11
tutor/plugins/v1.py
View File

@ -2,7 +2,7 @@ import importlib.util
import os
from glob import glob
import pkg_resources
import importlib_metadata
from tutor import hooks
@ -26,7 +26,7 @@ def _discover_entrypoint_plugins() -> None:
"""
with hooks.Contexts.PLUGINS.enter():
if "TUTOR_IGNORE_ENTRYPOINT_PLUGINS" not in os.environ:
for entrypoint in pkg_resources.iter_entry_points("tutor.plugin.v1"):
for entrypoint in importlib_metadata.entry_points(group="tutor.plugin.v1"):
discover_package(entrypoint)
@ -56,7 +56,7 @@ def discover_module(path: str) -> None:
spec.loader.exec_module(module)
def discover_package(entrypoint: pkg_resources.EntryPoint) -> None:
def discover_package(entrypoint: importlib_metadata.EntryPoint) -> None:
"""
Install a plugin from a python package.
"""
@ -68,10 +68,11 @@ def discover_package(entrypoint: pkg_resources.EntryPoint) -> None:
# Add plugin information
if entrypoint.dist is None:
raise ValueError(f"Could not read plugin version: {name}")
hooks.Filters.PLUGINS_INFO.add_item((name, entrypoint.dist.version))
dist_version = entrypoint.dist.version if entrypoint.dist else "Unknown"
hooks.Filters.PLUGINS_INFO.add_item((name, dist_version))
# Import module on enable
@hooks.Actions.PLUGIN_LOADED.add()
def load(plugin_name: str) -> None:
if name == plugin_name:
entrypoint.load()
importlib.import_module(entrypoint.value)

2
tutor/templates/apps/caddy/Caddyfile
View File

@ -37,6 +37,8 @@
reverse_proxy {args.0} {
header_up X-Forwarded-Port {{ 443 if ENABLE_HTTPS else 80 }}
}
{{ patch("caddyfile-proxy")|indent(4) }}
}
{{ LMS_HOST }}{$default_site_port}, {{ PREVIEW_LMS_HOST }}{$default_site_port} {

2
tutor/templates/apps/openedx/settings/partials/common_all.py
View File

@ -73,7 +73,7 @@ CACHES = {
},
"course_structure_cache": {
"KEY_PREFIX": "course_structure",
"TIMEOUT": 7200,
"TIMEOUT": 604800, # 1 week
"BACKEND": "django_redis.cache.RedisCache",
"LOCATION": "redis://{% if REDIS_USERNAME and REDIS_PASSWORD %}{{ REDIS_USERNAME }}:{{ REDIS_PASSWORD }}{% endif %}@{{ REDIS_HOST }}:{{ REDIS_PORT }}/{{ OPENEDX_CACHE_REDIS_DB }}",
},

7
tutor/templates/apps/redis/redis.conf
View File

@ -39,3 +39,10 @@ auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
aof-load-truncated yes
aof-use-rdb-preamble yes
############################## MEMORY MANAGEMENT ################################
maxmemory 4gb
maxmemory-policy allkeys-lru
{{ patch("redis-conf") }}

7
tutor/templates/build/openedx/Dockerfile
View File

@ -49,8 +49,13 @@ RUN git config --global user.email "tutor@overhang.io" \
{%- if patch("openedx-dockerfile-git-patches-default") %}
# Custom edx-platform patches
{{ patch("openedx-dockerfile-git-patches-default") }}
{%- elif EDX_PLATFORM_VERSION == "master" %}
# Patches in nightly node
{%- else %}
# Patch edx-platform
# Patches in non-nightly mode
# Prevent course structure cache infinite growth
# https://github.com/openedx/edx-platform/pull/34210
RUN curl -fsSL https://github.com/openedx/edx-platform/commit/ad201cd664b6c722cbefcbda23ae390c06daf621.patch | git am
{%- endif %}
{# Example: RUN curl -fsSL https://github.com/openedx/edx-platform/commit/<GITSHA1>.patch | git am #}

4
tutor/templates/config/defaults.yml
View File

@ -22,7 +22,7 @@ DOCKER_IMAGE_MONGODB: "docker.io/mongo:4.4.25"
DOCKER_IMAGE_MYSQL: "docker.io/mysql:8.1.0"
DOCKER_IMAGE_PERMISSIONS: "{{ DOCKER_REGISTRY }}overhangio/openedx-permissions:{{ TUTOR_VERSION }}"
# https://hub.docker.com/_/redis/tags
DOCKER_IMAGE_REDIS: "docker.io/redis:7.2.1"
DOCKER_IMAGE_REDIS: "docker.io/redis:7.2.4"
# https://hub.docker.com/r/devture/exim-relay/tags
DOCKER_IMAGE_SMTP: "docker.io/devture/exim-relay:4.96-r1-0"
EDX_PLATFORM_REPOSITORY: "https://github.com/openedx/edx-platform.git"
@ -59,7 +59,7 @@ OPENEDX_LMS_UWSGI_WORKERS: 2
OPENEDX_MYSQL_DATABASE: "openedx"
OPENEDX_MYSQL_USERNAME: "openedx"
# the common version will be automatically set to "master" in the nightly branch
OPENEDX_COMMON_VERSION: "open-release/quince.1"
OPENEDX_COMMON_VERSION: "open-release/quince.3"
OPENEDX_EXTRA_PIP_REQUIREMENTS: []
MYSQL_HOST: "mysql"
MYSQL_PORT: 3306

4
tutor/templates/dev/docker-compose.yml
View File

@ -6,10 +6,6 @@ x-openedx-service:
stdin_open: true
tty: true
volumes:
# Settings & config
- ../apps/openedx/settings/lms:/openedx/edx-platform/lms/envs/tutor:ro
- ../apps/openedx/settings/cms:/openedx/edx-platform/cms/envs/tutor:ro
- ../apps/openedx/config:/openedx/config:ro
# theme files
- ../build/openedx/themes:/openedx/themes

43
tutor/templates/jobs/init/mounted-directories.sh Normal file
View File

@ -0,0 +1,43 @@
# The initialization job contains various re-install operations needed to be done
# on mounted directories (edx-platform, /mnt/*xblock, /mnt/<edx-ora, search, enterprise>)
# 1. /mnt/*
# Whenever xblocks or other installable packages are mounted, during the image build, they are copied over to container
# and installed. This results in egg_info generation for the mounted directories. However, the egg_info is not carried
# over to host. When the containers are launched, the host directories without egg_info are mounted on runtime
# and disappear from pip list.
#
# 2. edx-platform
# When a new local copy of edx-platform is bind-mounted, certain build
# artifacts from the openedx image's edx-platform directory are lost.
# We regenerate them here.
for mounted_dir in /mnt/*; do
if [ -f $mounted_dir/setup.py ] && ! ls $mounted_dir/*.egg-info >/dev/null 2>&1 ; then
echo "Unable to locate egg-info in $mounted_dir"
pip install -e $mounted_dir
fi
done
if [ -f /openedx/edx-platform/bindmount-canary ] ; then
# If this file exists, then edx-platform has not been bind-mounted,
# so no build artifacts need to be regenerated.
echo "Using edx-platform from image (not bind-mount)."
echo "No extra setup is required."
exit
fi
echo "Performing additional setup for bind-mounted edx-platform."
set -x # Echo out executed lines
# Regenerate Open_edX.egg-info
pip install -e .
# Regenerate node_modules
npm clean-install
# Regenerate static assets.
openedx-assets build --env=dev
set -x
echo "Done setting up bind-mounted edx-platform."

26
tutor/templates/jobs/init/mounted-edx-platform.sh
View File

@ -1,26 +0,0 @@
# When a new local copy of edx-platform is bind-mounted, certain build
# artifacts from the openedx image's edx-platform directory are lost.
# We regenerate them here.
if [ -f /openedx/edx-platform/bindmount-canary ] ; then
# If this file exists, then edx-platform has not been bind-mounted,
# so no build artifacts need to be regenerated.
echo "Using edx-platform from image (not bind-mount)."
echo "No extra setup is required."
exit
fi
echo "Performing additional setup for bind-mounted edx-platform."
set -x # Echo out executed lines
# Regenerate Open_edX.egg-info
pip install -e .
# Regenerate node_modules
npm clean-install
# Regenerate static assets.
openedx-assets build --env=dev
set -x
echo "Done setting up bind-mounted edx-platform."