6
0
mirror of https://github.com/ChristianLight/tutor.git synced 2024-12-13 14:43:03 +00:00
Commit Graph

41 Commits

Author SHA1 Message Date
Braden MacDonald
b3879e13b6 docs: update ARM tutorial 2023-06-23 16:12:25 +02:00
Régis Behmo
947b37524f feat: auto build "openedx-dev" on "dev launch"
To achieve that, we introduce a new IMAGES_BUILD_REQUIRED filter.
2023-06-14 21:08:49 +02:00
Régis Behmo
2a21b2adf3 feat: images build openedx-dev
We no longer run `docker-compose up --build`. Instead, users are
encouraged to build the "openedx-dev" Docker image.
2023-06-14 21:08:49 +02:00
Régis Behmo
18ce1f2fe4 feat: persistent bind-mounts
This is an important change, where we get remove the previous `--mount`
option, and instead opt for persistent bind-mounts.

Persistent bind mounts have several advantages:
- They make it easier to remember which folders need to be bind-mounted.
- Code is *much* less clunky, as we no longer need to generate temporary
  docker-compose files.
- They allow us to bind-mount host directories *at build time* using the
  buildx `--build-context` option.
- The transition from development to production becomes much easier, as
  images will automatically be built using the host repo.

The only drawback is that persistent bind-mounts are slightly less
portable: when a config.yml file is moved to a different folder, many
things will break if the repo is not checked out in the same path.

For instance, this is how to start working on a local fork of
edx-platform:

    tutor config save --append MOUNTS=/path/to/edx-platform

And that's all there is to it. No, this fork will be used whenever we
run:

    tutor images build openedx
    tutor local start
    tutor dev start

This change is made possible by huge improvements in the build time
performance. These improvements make it convenient to re-build Docker
images often.

Related issues:
https://github.com/openedx/wg-developer-experience/issues/71
https://github.com/openedx/wg-developer-experience/issues/66
https://github.com/openedx/wg-developer-experience/issues/166
2023-06-14 21:08:49 +02:00
Régis Behmo
b3c3c4a2cc feat: upgrade to Palm
Among other changes: ORA2 file uploads were stored in a folder named
"SET-ME-PLEASE (ex.  bucket-name)" (sigh). With this change, the folder
should be automatically renamed to "openedxuploads". This issue has been
occuring since June 2019... (sigh²)

Close #707
2023-06-14 19:40:58 +02:00
Régis Behmo
f042ca9b90 docs: fix ENV_PATCHES example 2023-05-16 12:03:38 +02:00
Braden MacDonald
d7ed08110f docs: Update ARM64 tutorial 2023-04-13 15:03:57 +02:00
Régis Behmo
71ed7a8618 feat: refactor hooks API for simplification
The hooks API had several issues which are summarized in this comment:
https://github.com/openedx/wg-developer-experience/issues/125#issuecomment-1313553526

1. "consts" was a bad name
2. "hooks.filters" and "hooks.Filters" could easily be confused
3. docs made it difficult to understand that plugin developers should use the catalog

To address these issues, we:

1. move "consts.py" to "catalog.py"
2. Remove "hooks.actions", "hooks.filters", "hooks.contexts" from the API.
3. re-organize the docs and give better usage examples in the catalog.

This change is a partial fix for https://github.com/openedx/wg-developer-experience/issues/125
2023-01-31 10:17:58 +01:00
Abhiram Kuchibhotla
d7bc3370df
docs: update the Podman instructions
Close #771.
2023-01-06 17:46:33 +01:00
Régis Behmo
e4e9cd8197 Merge branch 'master' into nightly 2022-11-29 10:28:11 +01:00
Régis Behmo
4a6896d892 docs: adjust web proxy tutorial 2022-11-29 09:49:57 +01:00
Daniel Herrmann
bb137c16d6 Added more info to required headers and a pitfall regarding misleading wording in the quickstart dialogue 2022-11-29 09:49:57 +01:00
Régis Behmo
c06ac5b020 Merge branch 'master' into nightly 2022-11-24 17:27:47 +01:00
Régis Behmo
6069f91cce docs: fix nightly typo
Co-authored-by: Adolfo R. Brandes <arbrandes@tcril.org>
2022-11-24 16:15:15 +01:00
Régis Behmo
c886b72ae7 docs: migrate to scriv to manage changelog
Changelog management was starting to be a hassle:

- there were conflicts every time a PR was merged
- there were conflicts every time we merged the nightly branch in the new
  release branch, or vice versa.

Now, all changelog entries are stored as separate files in changelog.d,
including nightly. Nightly entries will be collected for every major release.
2022-11-24 16:15:15 +01:00
Régis Behmo
16e6131f96 feat: pluggable local/dev/k8s do <job> commands
We introduce a new filter to implement custom commands in arbitrary containers.
It becomes easy to write convenient ad-hoc commands that users will
then be able to run either on Kubernetes or locally using a documented CLI.

Pluggable jobs are declared as Click commands and are responsible for
parsing their own arguments. See the new CLI_DO_COMMANDS filter.

Close https://github.com/overhangio/2u-tutor-adoption/issues/75
2022-11-15 09:46:08 +01:00
Régis Behmo
883b358427 Merge branch 'master' into nightly 2022-10-03 15:13:53 +02:00
Michael Wheeler
fd037fcb4d Remove warning about building tutor-mfe on ARM systems 2022-10-03 14:15:24 +02:00
Carlos Muniz
fe901ab9de feat: deprecate "quickstart" and rename to "launch"
`quickstart` is being renamed to `launch` and deprecated in favor of
using `launch`. The `quickstart` function temporarily aliases to
`launch`. Further mentions of `quickstart` have been changed to
reference `launch` instead.

We are indicating that this change is breaking 💥 to encourage people to
migrate their scripts right away!
2022-09-30 12:05:35 +02:00
Régis Behmo
4b015bdc4e docs: move forum to discuss.openedx.org
Now that Tutor is the official community installation for Open edX, it no
longer makes sense to host a forum that is separate from the general Open edX
forum. Moving conversations there will encourage cross-communication between
projects and maintainers. This change is part of a larger overhaul described in
this Tutor enhancement proposal (TEP):
https://discuss.overhang.io/t/tep-rethinking-the-tutor-maintainers-program/2724

In the future, plugin maintainers should point their users to the Open edX
forum as well. They are encouraged to create dedicated "tutor-pluginnname" tags
on the forum and to set their notification level to "watching".
2022-05-23 09:48:19 +02:00
Kyle McCormick
57108bdf76 docs: specify root username in mysql dump instructions
The default user in the mysql container is 'mysql',
so the `mysql` command tries to use the 'mysql' MySQL user by default.

But, in the MySQL dump instructions,
we are providing the MySQL root user's password, so we need
to specify `MYSQL_ROOT_USERNAME` as the MySQL user when invoking `mysql`.
2022-05-20 07:43:56 +02:00
Régis Behmo
7d20329894 feat: local stop on dev start (and vice versa)
Running `local start` while a dev platform is still running is a common sourse
of mistakes. Here we introduce a new action to automatically stop local and dev
projects whenever a project with a different name is started.
2022-04-24 09:56:12 +02:00
Kyle McCormick
5a81b6a6cb feat: deprecate runserver in favor of start
`tutor dev runserver` will be removed in a future release.
Developers are encouraged to use `tutor dev start` instead,
which is more flexible and provides a consistent interface
with `tutor local start`.

As part of this deprecation, we enable the `tty` and
`stdin_open` options on development docker-compose
services. This will allow developers to use `start`
for breakpoint debugging, which was previously only
availble via `runserver`. Several parallel PRs have
been merged in order to make the same change in the
development services of the official plugins.

Although `start` does not support the `--volume` option,
it supports a more-powerful `--mount` option. So, where
developers previously used:

    tutor dev runserver --volume ...

to bind-mount host directories, they should now use:

    tutor dev start --mount ...

Resolves https://github.com/overhangio/2u-tutor-adoption/issues/61
2022-04-21 15:00:54 +02:00
Régis Behmo
5de7bec029 depr: drop the possibility of overriding edx-platform settings
Previously, it was possible to override settings by defining the
TUTOR_EDX_PLATFORM_SETTINGS environment variable. But let's face it:

- It was not very well supported.
- It was poorly explained.
- It was not very useful.
- It causes unnecessary code complexity.

For these reasons, we drop that feature.
2022-04-15 15:37:56 +02:00
Régis Behmo
15b219e235 feat: migrate to plugins.v1 with filters & actions
This is a very large refactoring which aims at making Tutor both more
extendable and more generic. Historically, the Tutor plugin system was
designed as an ad-hoc solution to allow developers to modify their own
Open edX platforms without having to fork Tutor. The plugin API was
simple, but limited, because of its ad-hoc nature. As a consequence,
there were many things that plugin developers could not do, such as
extending different parts of the CLI or adding custom template filters.

Here, we refactor the whole codebase to make use of a generic plugin
system. This system was inspired by the Wordpress plugin API and the
Open edX "hooks and filters" API. The various components are added to a
small core thanks to a set of actions and filters. Actions are callback
functions that can be triggered at different points of the application
lifecycle. Filters are functions that modify some data. Both actions and
filters are collectively named as "hooks". Hooks can optionally be
created within a certain context, which makes it easier to keep track of
which application created which callback.

This new hooks system allows us to provide a Python API that developers
can use to extend their applications. The API reference is added to the
documentation, along with a new plugin development tutorial.

The plugin v0 API remains supported for backward compatibility of
existing plugins.

Done:
- Do not load commands from plugins which are not enabled.
- Load enabled plugins once on start.
- Implement contexts for actions and filters, which allow us to keep track of
  the source of every hook.
- Migrate patches
- Migrate commands
- Migrate plugin detection
- Migrate templates_root
- Migrate config
- Migrate template environment globals and filters
- Migrate hooks to tasks
- Generate hook documentation
- Generate patch reference documentation
- Add the concept of action priority

Close #499.
2022-04-15 15:30:54 +02:00
Kyle McCormick
3ca4522f68 docs: update Nightly docs to reference [full] installation
The full installation will include all the plugins that
come bundled with Tutor stable. This is made possible by
a recent change to Tutor Nightly
(https://github.com/overhangio/tutor/pull/626).
2022-04-15 09:20:21 +02:00
Emad Rad
a25ae73031 minor typos fixed 2022-03-29 08:44:33 +02:00
Régis Behmo
8e052f703f docs: clarify image building on arm64
Close #590.
2022-03-17 10:38:55 +01:00
Jayram Nai
37be2bf122 docs: fixed typo 2022-01-08 17:58:26 +01:00
Braden MacDonald
dbb79c0fa0 docs: Address review comments 2022-01-03 06:56:58 +01:00
Braden MacDonald
c240aec711 docs: Document how to run Tutor on ARM-based systems 2022-01-03 06:56:58 +01:00
Régis Behmo
7c157eccd5 feat: upgrade to Maple
- A shared cookie domain between lms and cms is no longer recommended:
https://github.com/edx/edx-platform/blob/master/docs/guides/studio_oauth.rst
- refactor: clean mounted data folder in lms/cms. In Lilac, the
bind-mounted lms/data and cms/data folders are a mess because new
folders are created there for every new course organisation.  These
folders are empty. As far as we know they are useless... With this
change we move these folders to a dedicated "modulestore" subdirectory;
which corresponds better to the initial intent of the fs_root setting.
- fix: frontend failure during login to the lms. See:
https://github.com/openedx/build-test-release-wg/issues/104
- feat: move all forum-related code to a dedicated plugin. Forum is an
optional feature, and as such it deserves its own plugin. Starting from
Maple, users will be able to install the forum from
https://github.com/overhangio/tutor-forum/
- migrate from DCS_* session cookie settings to SESSION_*. That's
because edx-platform no longer depends on django-cookies-samesite. Close
https://github.com/openedx/build-test-release-wg/issues/110
- get rid of tons of deprecation warnings in the lms/cms
- feat: make it possible to point to themed assets. Cherry-picking this
change makes it possible to point to themed assets with a theme-agnostic
url, notably from MFEs.
- Install all official plugins as part of the `tutor[full]` package.
- Don't print error messages about loading plugins during autocompletion.
- Prompt for image building when upgrading from one release to the next.
- Add `tutor local start --skip-build` option to skip building Docker images.

Close #450.
Close #545.
2021-12-20 21:21:36 +01:00
Régis Behmo
62ddc01cdc feat: k8s scale command + tutorial
- Add a `tutor k8s scale lms 11` command
- Create a "Running Open edX at scale" tutorial
2021-12-20 21:19:10 +01:00
Régis Behmo
01b58d9d75 feat: run all services as unprivileged containers
With this change, containers are no longer run as "root" but as unprivileged
users. This is necessary in some environments, notably some Kubernetes
clusters.

To make this possible, we need to manually fix bind-mounted volumes in
docker-compose. This is pretty much equivalent to the behaviour in Kubernetes,
where permissions are fixed at runtime if the volume owner is incorrect. Thus,
we have a consistent behaviour between docker-compose and Kubernetes.

We achieve this by bind-mounting some repos inside "*-permissions" services.
These services run as root user on docker-compose and will fix the required
permissions, as per build/permissions/setowner.sh These services simply do not
run on Kubernetes, where we don't rely on bind-mounted volumes. There, we make
use of Kubernete's built-in volume ownership feature.

With this change, we get rid of the "openedx-dev" Docker image, in the sense
that it no longer has its own Dockerfile. Instead, the dev image is now simply
a different target in the multi-layer openedx Docker image. This makes it much
faster to build the openedx-dev image.

Because we declare the APP_USER_ID in the dev/docker-compose.yml file, we need
to pass the user ID from the host there. The only way to achieve that is with a
tutor config variable. The downside of this approach is that the
dev/docker-compose.yml file is no longer portable from one machine to the next.
We consider that this is not such a big issue, as it affects the development
environment only.

We take this opportunity to replace the base image of the "forum" image. There
is now no need to re-install ruby inside the image. The total image size is
only decreased by 10%, but re-building the image is faster.

In order to run the smtp service as non-root, we switch from namshi/smtp to
devture/exim-relay. This change should be backward-compatible.

Note that the nginx container remains privileged. We could switch to
nginxinc/nginx-unprivileged, but it's probably not worth the effort, as we are
considering to get rid of the nginx container altogether.

Close #323.
2021-12-20 21:19:10 +01:00
Régis Behmo
231bbbfe99 feat: get rid of the nginx container and services
Nginx and Caddy performed duplicate tasks. It was decided to get rid of
the nginx container, for simplification. This is a breaking change for
plugin developers. Also, applications that collect nginx logs will have
to be modified.

See:
- Corresponding TEP: https://discuss.overhang.io/t/tep-get-rid-of-the-nginx-container/2024
- the prior discussion: https://discuss.overhang.io/t/why-caddy-nginx/1952
2021-12-20 21:19:10 +01:00
Régis Behmo
d4fe1260c1 docs: add tutorial on using Google Mail as an SMTP server
The tutorial comes from the feedback of multiple users, including:
https://discuss.overhang.io/t/google-smtp-not-working/2143
2021-11-25 12:25:23 +01:00
Régis Behmo
030d56f9af docs: nightly development workflow 2021-10-14 12:59:57 +02:00
Régis Behmo
33ca30d6c3 goodbye "edge" hello "nightly"!
In conversations with edX, we learned that the name "edge" had negative
undertones for historical reasons. Thus, we switch to "nightly", which means
pretty much the same thing.
2021-10-14 12:59:57 +02:00
Régis Behmo
a6af8a4e0f docs: add tutorial on running tutor edge 2021-10-14 12:59:57 +02:00
Régis Behmo
b57c65440a docs: move podman install to its own tutorial
<rant>I attempted to actually run Tutor with Podman and I was sorely disappointed.
The only reliable source of docs that I found concerning the integration with
docker-compose is this blog post:
https://www.redhat.com/sysadmin/podman-docker-compose
There are no other official docs 😓

1. The instructions given in the blog post don't work out of the box. Launching
the podman service failed altogether on Ubuntu 20.04 and 20.10. It worked on
CentOS 8, but some parameters need to changed, such as the docker socket path.
2. After I got the podman service working, I managed to get an Open edX
platform running with tutor, but with the root user. Then, containers
complained that they could not write data to the bind-mounted volumes. I
attempted to run as a non-root user, and discovered that the podman socket is
only readable by root. This should explain why all commands from that blog post
are prefixed by sudo.

Long story short, I was hoping to update the tutorial. Instead, I'm just moving
it for the sake of better organisation. For the life of me, I do not understand
why some people would want to run Podman instead of Docker. Bad documentation
is an immediate turn-off for me. From my perspective, podman is mostly an
overblown marketina stunt.</rant>
2021-10-14 12:59:57 +02:00
Régis Behmo
e14f660cb1 docs: reorganize local guides in tutorials
There is too much information in each of the local/k8s/dev docs pages. The
"guides" that are listed in each one of those pages are moved either to "common
tasks" or to a dedicated "tutorials" section. This paves the way for more
comprehensive tutorials, where we describe how to run the latest master
branches of Open edX.

I am well aware that, as they stand, the tutorials are of poor quality and
should be rewritten. This is a task for another day/commit. For now, we only
move the contents to a separate part of the docs.

Also, we should add a "reference" section to the docs, where we add the result
of `tutor <subcommand> --help`.
2021-10-14 12:59:57 +02:00