7
0
mirror of https://github.com/ChristianLight/tutor.git synced 2024-06-15 20:12:20 +00:00
Commit Graph

50 Commits

Author SHA1 Message Date
Kyle McCormick
c273e7f105
feat!: assume BuildKit is available 2023-10-27 10:54:23 +02:00
Régis Behmo
3ab0dcb9e6 depr: templated hooks
Templated hooks we almost completely useless, so we get rid of them.
This allows us to get rid entirely of hook names and hook indexes, which
makes the whole implementation much simpler. Hook removal (with
`clear_all`) is achieved thanks to weak references.
2023-06-14 21:08:49 +02:00
Régis Behmo
7972a75915 feat: leverage RUN --mount for faster image building
We make use of the Docker build cache to install python and nodejs
requirements faster in the case of repeated builds.

This feature is only possible for users of BuildKit, so we detect
whether `docker buildx` is available at runtime.

We do not make use of `COPY --link` because the `--link` option is
incompatible with `--chown=app:app`:
https://github.com/docker/buildx/issues/1408

For reference, see:

https://www.docker.com/blog/dockerfiles-now-support-multiple-build-contexts/
https://docs.docker.com/engine/reference/commandline/buildx_build/#build-context
2023-06-14 21:08:49 +02:00
Régis Behmo
39a8b46854 depr: drop support for docker-compose
Instead, the compose plugin must be installed.
We deprecate docker-compose because v1 is not supported starting from
the end of June 2023.

See "evolution of compose": https://docs.docker.com/compose/compose-v2/
2023-06-14 19:40:58 +02:00
Régis Behmo
11b2091e32 depr: halt compatibility with python 3.7 2023-06-14 19:40:58 +02:00
Régis Behmo
b3c3c4a2cc feat: upgrade to Palm
Among other changes: ORA2 file uploads were stored in a folder named
"SET-ME-PLEASE (ex.  bucket-name)" (sigh). With this change, the folder
should be automatically renamed to "openedxuploads". This issue has been
occuring since June 2019... (sigh²)

Close #707
2023-06-14 19:40:58 +02:00
Maria Fernanda Magallanes Zubillaga
40e5c8da6b refactor: move format_table function to utils 2023-02-28 09:45:11 +01:00
Régis Behmo
fa318a64ce feat: plugin indexes
We implement this TEP: https://discuss.openedx.org/t/tutor-enhancement-proposal-tep-plugin-indices/8182

With plugin indexes, tutor users can install and upgrade plugins directly from indexes:

    tutor plugins install ecommerce
    tutor plugins index add contrib
    tutor plugins install codejail
    tutor plugins upgrade all

This change has been long in the coming \o/
2023-02-10 10:06:38 +01:00
Régis Behmo
16e6131f96 feat: pluggable local/dev/k8s do <job> commands
We introduce a new filter to implement custom commands in arbitrary containers.
It becomes easy to write convenient ad-hoc commands that users will
then be able to run either on Kubernetes or locally using a documented CLI.

Pluggable jobs are declared as Click commands and are responsible for
parsing their own arguments. See the new CLI_DO_COMMANDS filter.

Close https://github.com/overhangio/2u-tutor-adoption/issues/75
2022-11-15 09:46:08 +01:00
Régis Behmo
f95c394e38 fix: print proper command on check_output
Note that the `check_output` command is mostly unused.
2022-10-26 16:46:04 +02:00
Régis Behmo
a2a3c022b8 fix: bind-mount in dev-specific services
The -m/--mount option makes it possible to bind-mount volumes at runtime. The
volumes are declared in a local/docker-compose.tmp.yml file. The problem with
this approach is when we want to bind-mount a volume to a service which is
specific to the dev context. For instance: the "learning" service when the MFE
plugin is enabled.

In such a case, starting the service triggers a call to `docker-compose stop`
in the local context. This call fails because the "learning" service does not
exist in the local context. Note that this issue only seems to occur with
docker-compose v1.

To resolve this issue, we create two additional filters for
the dev context, which emulate the behaviour of the local context. With this approach, we convert the -m/--mount arguments right after they are parsed. Because they are parsed just once, we can get rid of the de-duplication logic initially introduced with the COMPOSE_CLI_MOUNTS context.

Close #711. Close also https://github.com/overhangio/tutor-mfe/issues/57.
2022-07-29 19:53:02 +02:00
Kyle McCormick
8345b7ab93 temp: shim shlex.join for Python 3.7 compatability 2022-07-25 22:46:53 +02:00
Kyle McCormick
6a3138152f fix: accurately log shell commands when running jobs
Whenever Tutor executes a shell command, it logs out said
command in order to aid in end user understanding/debugging.

In some cases (notably, when running jobs in containers)
the logged command was not accurately quoted. The command
was run correctly, because it was passed in pieces to
``subprocess.Popen``, which correctly joins the pieces together
into a valid POSIX shell command; however, the logged version
of the command was constructed by simply joining the pieces
with spaces. This usually works, but breaks down when running
complex shell commands with nested quoting.

This commit changes the logging to use ``shlex.join``, which
joins command pieces together in a POSIX-compliant way,
presumably the same way as ``subprocess.Popen``.

Example:

    tutor local importdemocourse

runs the shell command:

    docker-compose -f /home/kyle/.local/share/tutor/env/local/docker-compose.yml -f /home/kyle/.local/share/tutor/env/local/docker-compose.prod.yml -f /home/kyle/.local/share/tutor/env/local/docker-compose.tmp.yml --project-name tutor_local -f /home/kyle/.local/share/tutor/env/local/docker-compose.jobs.yml -f /home/kyle/.local/share/tutor/env/local/docker-compose.jobs.tmp.yml run --rm cms-job sh -e -c 'echo "Loading settings $DJANGO_SE... (several more script lines) ...eindex_course --all --setup'

but the logged shell command was:

    docker-compose -f /home/kyle/.local/share/tutor/env/local/docker-compose.yml -f /home/kyle/.local/share/tutor/env/local/docker-compose.prod.yml -f /home/kyle/.local/share/tutor/env/local/docker-compose.tmp.yml --project-name tutor_local -f /home/kyle/.local/share/tutor/env/local/docker-compose.jobs.yml -f /home/kyle/.local/share/tutor/env/local/docker-compose.jobs.tmp.yml run --rm cms-job sh -e -c echo "Loading settings $DJANGO_SE... (several more script lines) ...eindex_course --all --setup

which will not run if copied and pasted back into the
user's terminal, as the importdemocourse shell script is unquoted.
2022-07-25 22:46:53 +02:00
Régis Behmo
b13d4da1b1 fix: @lru_cache python 3.7 compatibility
In Python 3.7, which is used in the Tutor CI, the lru_cache needs an argument:
https://docs.python.org/3.7/library/functools.html#functools.lru_cache
On GitHub, CI did not catch this error because it is running Python 3.8.
2022-06-22 11:28:27 +02:00
iamcristye@outlook.com
a4aa01987b improvement: compatibility with Docker Compose subcommand
closes #641
2022-06-22 11:23:39 +02:00
Régis Behmo
79f14b7e7e chore: fix various linting warnings in f-strings and docs 2022-02-25 14:50:36 +01:00
Régis Behmo
43d5da83e4 fix: utils tests on macOS
test_utils tests were failing on macOS when the settings file was properly
defined and present.

Close #560.
2022-01-08 18:48:50 +01:00
alex.soh
72843c06f9 refactor: add code coverage, cover CLI commands with tests 2022-01-04 13:40:33 +01:00
Régis Behmo
78117d16f2 chore: get rid of outdated pycryptodome ugly patch
This patch is no longer required now that the fix has been merged upstream, in
3.10.3: https://github.com/Legrandin/pycryptodome/issues/506
2021-10-25 20:19:27 +02:00
Peter Parkkali
a095a6fbc7 fix: Change memory allocation error into a warning in 'local quickstart' 2021-10-25 20:17:38 +02:00
Peter Parkkali
2549aef4dc fix: require at least 4 GB RAM on macOS for 'local quickstart' only
Limits the memory chek to the 'local quickstart' command, makes error
handling more accurate and adds warning messages for some conditions.
Also adds a mention of this in troubleshooting.rst.
2021-10-25 20:17:38 +02:00
Peter Parkkali
fb2aeefd91 fix: require at least 4 GB RAM on macOS for local commands
Adds a check in the 'local' command group that requires at least
4 GB of RAM to be allocated to Docker when running any of the
local subcommands on macOS. This addresses a common issue where
Docker's default setting (2 GB) causes startup to crash with
misleading error messages.
2021-10-25 20:17:38 +02:00
Régis Behmo
b651ebc3c4 fix: segmentation fault on Mac OS M1 (Big Sur)
On Mac OS M1 (ARM processor), when the GMP library is available, pycryptodome
crashes with a segmentation fault during the generation of the RSA keys.

Upstream issue: https://github.com/Legrandin/pycryptodome/issues/506
Upstream fix: https://github.com/Legrandin/pycryptodome/pull/541

Close #473.
2021-09-17 10:44:52 +02:00
Régis Behmo
b8394471ec feat: catch errors when writing a file where a directory exists
This error sometimes happens when developing new plugins.
2021-03-15 23:26:38 +01:00
Régis Behmo
0a670d7ead refactor: add type annotations
Annotations were generated with pyannotate:
https://github.com/dropbox/pyannotate

We are running in strict mode, which is awesome!

This affects a large part of the code base, which might be an issue for
people running a fork of Tutor. Nonetheless, the behavior should not be
affected. If anything, this process has helped find and resolve a few
type-related bugs. Thus, this is not considered as a breaking change.
2021-03-15 21:46:55 +01:00
Régis Behmo
de872282b2 fix: minor edge case in long_to_base64 utility function.
This was detected by mypy.
2021-02-25 09:54:46 +01:00
Régis Behmo
a4051302ad Don't even try to detect user properties on Windows 2020-11-19 11:32:17 +01:00
Régis Behmo
be82964bdc Fix utils.check_output function
This was printing meaningless warning messages.
2020-11-19 11:32:17 +01:00
Régis Behmo
21231a15d0 Minor simplification for checking root usage 2020-11-16 12:17:15 +01:00
Régis Behmo
b2fdaf604e Fix platform checking and env saving on Windows 2020-11-12 12:56:18 +01:00
Régis Behmo
6c4d9619fd Fix user ID checking under windows
This should make tutor portable on Windows. Well, if the `id -u` command
is supported at least...

See: https://github.com/overhangio/tutor/issues/381
2020-11-07 17:48:20 +01:00
Régis Behmo
6848253b8e Switch from crypt to bcrypt for htpasswd generation
This is more portable, as described here:
https://github.com/overhangio/tutor/issues/381
2020-11-07 14:46:53 +01:00
Régis Behmo
d978401155 Better error management in execute internal function 2020-10-30 18:20:19 +01:00
Régis Behmo
4d6de0138a v10.0.0 Upgrade to Juniper (2020-06-15)
Here, we upgrade the Open edX platform from Ironwood to Juniper. This
upgrade does not come with many feature changes, but there are many
technical improvements under the hood:

- Upgrade from Python 2.7 to 3.5
- Upgrade from Mongodb v3.2 to v3.6
- Upgrade Ruby to 2.5.7

We took the opportunity to completely rething the way locally running
platforms should be accessed for testing purposes. It is no longer
possible to access a running platform from http://localhost and
http://studio.localhost. Instead, users should access
http://local.overhang.io and https://studio.local.overhang.io. This
drastically simplifies internal communication between Docker containers.

To upgrade, users should simply run:

    tutor local quickstart

For Kubernetes platform, the upgrade process is outlined when running:

    tutor k8s upgrade --from=ironwood
2020-06-15 10:19:07 +02:00
Régis Behmo
411327662e Add encrypt template filter
This is convenient for htpasswd-based authentication to nginx, for
instance.
2020-04-04 18:22:15 +02:00
Régis Behmo
b72da818ea Minor formatting 2020-03-27 10:17:36 +01:00
Régis Behmo
e521333b75 Fix tls certificate generation in cron jobs
tls renewal and generation was failing in cron jobs because of "The
input device is not a TTY" errors. This is because the "-it" docker
option does not work when a tty is not available.
2020-03-27 10:05:40 +01:00
Régis Behmo
c89a5d3c3a Minor black formatting issue 2019-11-22 09:23:59 +01:00
Régis Behmo
993694909a Fix forum depends_on when data services are disabled
When both mongodb and elasticsearch were not activated, there was a
syntax error in the local docker-compose.yml file.

Close #266
2019-11-22 09:21:59 +01:00
Régis Behmo
6a68c4cc20 Progress on the plugins/k8s front
This commit introduces many changes:
- a fully functional minio plugin for local installation
- an almost-functional native k8s deployment
- a new way to process configuration, better suited to plugins

There are still many things to do:
- get rid of all the TODOs
- get a fully functional minio plugin for k8s
- add documentation for pluginso
- ...
2019-06-07 22:49:45 +02:00
Régis Behmo
334f3e720e Towards a stable Kubernetes integration
Missing features:
- https certificates
- xqueue
- lms/cms workers

Moreover, we scalability issues due to the uploaded file storage in the
lms/cms. To address this issue we need to develop the MinIO plugin so
that it becomes compatible with Open edX.

Close #126 #179 #187
2019-06-07 22:49:45 +02:00
Régis Behmo
407659ff06 Major refactoring of config module
Configuration loading was overly complex. Here, we simplify it
drastically with reasonable defaults. Hacky additional variables are
unncessary now that we use custom jinja2 filters.
2019-05-29 09:53:54 +02:00
Régis Behmo
f2f714ad23 Run code formatting on the entire code base 2019-05-05 12:30:20 +02:00
Régis Behmo
207229e16e Run pylint on code base 2019-04-25 09:36:18 +02:00
Régis Behmo
92c9132180 Move all yaml-related code to dedicated module 2019-03-24 14:34:50 -07:00
Régis Behmo
288dc12796 Fix yaml parsing of "null" 2019-03-24 14:27:00 -07:00
Régis Behmo
a7ab1c5ace Add missing CMS config (just in case)
I don't think this affects the CMS behaviour, but SESSION_COOKIE_DOMAIN
is used by the CMS, so rather be safe than sorry.
2019-03-23 16:15:51 -07:00
Régis Behmo
f45a24caea Easily configurable web proxy
A web proxy is a web server on the host that will forward request to the
tutor web server. We support Apache2 and Nginx.
2019-03-21 17:38:01 +01:00
Régis Behmo
e3b10b72f2 Load configuration values from environment
Configuration values can be loaded from the system environment by adding
a "TUTOR_" prefix.

Environment values supersede values from the user configuration file, so
that we can set values from the command line with "KEY=VAL tutor config
save --silent" even when KEY is already present in the user
configuration file.
2019-03-18 21:47:59 +01:00
Régis Behmo
4331bc5712 Tutor v3 complete rewrite
Replace all make commands by a single "tutor" binary. Environment and
data are all moved to ~/.tutor/local/share/tutor. We take the
opportunity to add a web UI and revamp the documentation.

This is a complete rewrite.

Close #121.
Close #147.
2019-02-09 20:30:01 +01:00