6
0
mirror of https://github.com/ChristianLight/tutor.git synced 2024-11-15 17:47:13 +00:00
Commit Graph

1626 Commits

Author SHA1 Message Date
Overhang.IO
31fdcc60af Merge remote-tracking branch 'origin/master' into nightly 2022-07-29 18:40:39 +00:00
Régis Behmo
66bef2e14f v14.0.4 (2022-07-29)
- [Feature] Add the `-m/--mount` option to `tutor dev quickstart`.
- [Bugfix] Fix `tutor dev start -m /path/to/frontend-app-learning` by introducing dev-specific `COMPOSE_DEV_TMP` and `COMPOSE_DEV_JOBS_TMP` filters (by @regisb).
- [Bugfix] Log the shell commands that Tutor executes more accurately. (by @kdmccormick)
- [Bugfix] `tutor dev quickstart` would fail under certain versions of docker-compose due to a bug in the logic that handled volume mounting. (by @kdmccormick)
- [Bugfix] The `tutor k8s start` command will succeed even when `k8s-override` and `kustomization-patches-strategic-merge` are not specified. (by @edazzocaisser)
- [BugFix] `kubectl wait` checks deployments instead of pods as it could hang indefinitely if there are extra pods in a broken state. (by @keithgg)
2022-07-29 19:54:15 +02:00
Régis Behmo
52cf0cc1ca feat: add -m/--mount option to dev quickstart 2022-07-29 19:53:02 +02:00
Régis Behmo
a2a3c022b8 fix: bind-mount in dev-specific services
The -m/--mount option makes it possible to bind-mount volumes at runtime. The
volumes are declared in a local/docker-compose.tmp.yml file. The problem with
this approach is when we want to bind-mount a volume to a service which is
specific to the dev context. For instance: the "learning" service when the MFE
plugin is enabled.

In such a case, starting the service triggers a call to `docker-compose stop`
in the local context. This call fails because the "learning" service does not
exist in the local context. Note that this issue only seems to occur with
docker-compose v1.

To resolve this issue, we create two additional filters for
the dev context, which emulate the behaviour of the local context. With this approach, we convert the -m/--mount arguments right after they are parsed. Because they are parsed just once, we can get rid of the de-duplication logic initially introduced with the COMPOSE_CLI_MOUNTS context.

Close #711. Close also https://github.com/overhangio/tutor-mfe/issues/57.
2022-07-29 19:53:02 +02:00
Overhang.IO
cb1b22f54e Merge remote-tracking branch 'origin/master' into nightly 2022-07-25 20:51:54 +00:00
Kyle McCormick
8345b7ab93 temp: shim shlex.join for Python 3.7 compatability 2022-07-25 22:46:53 +02:00
Kyle McCormick
6a3138152f fix: accurately log shell commands when running jobs
Whenever Tutor executes a shell command, it logs out said
command in order to aid in end user understanding/debugging.

In some cases (notably, when running jobs in containers)
the logged command was not accurately quoted. The command
was run correctly, because it was passed in pieces to
``subprocess.Popen``, which correctly joins the pieces together
into a valid POSIX shell command; however, the logged version
of the command was constructed by simply joining the pieces
with spaces. This usually works, but breaks down when running
complex shell commands with nested quoting.

This commit changes the logging to use ``shlex.join``, which
joins command pieces together in a POSIX-compliant way,
presumably the same way as ``subprocess.Popen``.

Example:

    tutor local importdemocourse

runs the shell command:

    docker-compose -f /home/kyle/.local/share/tutor/env/local/docker-compose.yml -f /home/kyle/.local/share/tutor/env/local/docker-compose.prod.yml -f /home/kyle/.local/share/tutor/env/local/docker-compose.tmp.yml --project-name tutor_local -f /home/kyle/.local/share/tutor/env/local/docker-compose.jobs.yml -f /home/kyle/.local/share/tutor/env/local/docker-compose.jobs.tmp.yml run --rm cms-job sh -e -c 'echo "Loading settings $DJANGO_SE... (several more script lines) ...eindex_course --all --setup'

but the logged shell command was:

    docker-compose -f /home/kyle/.local/share/tutor/env/local/docker-compose.yml -f /home/kyle/.local/share/tutor/env/local/docker-compose.prod.yml -f /home/kyle/.local/share/tutor/env/local/docker-compose.tmp.yml --project-name tutor_local -f /home/kyle/.local/share/tutor/env/local/docker-compose.jobs.yml -f /home/kyle/.local/share/tutor/env/local/docker-compose.jobs.tmp.yml run --rm cms-job sh -e -c echo "Loading settings $DJANGO_SE... (several more script lines) ...eindex_course --all --setup

which will not run if copied and pasted back into the
user's terminal, as the importdemocourse shell script is unquoted.
2022-07-25 22:46:53 +02:00
Keith Grootboom
55bc4c63fa fix: change kubectl wait to look at deployments not pods
When waiting for pods, it's possible that the deployment may be
complete but, because other pods may have been Evicted or Killed, the
wait wait condition completes.
2022-07-25 22:46:45 +02:00
Régis Behmo
8e188dc946 Merge branch 'master' into nightly 2022-07-25 16:13:05 +02:00
Kyle McCormick
0ae59a82a6
fix: avoid double-rendering mounts to docker-compose.tmp.yml (#669)
In certain code paths, such as in `tutor local quickstart`,
`process_mount_points` is called more than once in the same process,
causing mounts to be added to `COMPOSE_LOCAL[_JOBS]_TMP` redundantly.
As a result, docker-compose[.jobs].tmp.yml was occasionally being
rendered with duplicate volume specifiers. Some versions of Docker
Compose ignored this; other versions warned or threw an error.

In order to make `process_mount_points` tolerant to being called
multiple times, we wrap its volume-adding callbacks within a new
hooks context. This allows us to clear said hooks context every
time `process_mount_points` is called, essentially making the
function idempotent.

Co-authored-by: Régis Behmo <regis@behmo.com>
2022-07-25 15:58:49 +02:00
Carlos Muniz
d27e8d5ba7
docs: add missing config parameters from defaults.yaml
Document config parameters from defaults.yaml that were missing from the docs.


Co-authored-by: Carlos Muniz <cmuniz@trcil.org>
2022-07-25 15:43:40 +02:00
Ed D'Azzo
2e41de1af9 Added a record of the bugfix to the CHANGELOG 2022-07-25 15:25:25 +02:00
Ed D'Azzo
5e81a754c4 Fixed or syntax 2022-07-25 15:25:25 +02:00
Ed D'Azzo
67358c6b7a Omit patchesStrategicMerge when there are no custom resources 2022-07-25 15:25:25 +02:00
Kyle McCormick
9d1ce4717b docs: update test-running guide with new top-level xmodule package
edx-platform's ./common/lib/xmodule/xmodule folder has been moved to
./xmodule. The test-running instructions needed to be updated in order
to account for this new top-level folder.

For context on the edx-platform change, see:
https://discuss.openedx.org/t/breaking-apart-edx-platforms-common-lib-folder
2022-07-14 14:46:12 +02:00
David Ormsbee
bb907b4de3 v14.0.3 (2022-07-09)
- [Bugfix] Build openedx-dev Docker image even when the host user is root, for instance on Windows. (by @regisb)
- [Bugfix] Patch nutmeg.1 release with [LTI 1.3 fix](https://github.com/openedx/edx-platform/pull/30716). (by @ormsbee)
- [Improvement] Make it possible to override k8s resources in plugins using `k8s-override` patch. (by @foadlind)
2022-07-09 19:01:58 +02:00
Régis Behmo
5b5700eff4 fix: build openedx-dev image when host user is root
Sometimes, the host user is root: this may happen when tutor is run with
"sudo" (which is not recommended) or on Windows. In such cases, building
the image should not fail, but default to a reasonable user. Also, when
we pass an invalid APP_USER_ID as a build arg, then we should fail with
an explicit message.

See this conversation:
https://discuss.overhang.io/t/problem-with-dev-image-build-useradd-uid-0-is-not-unique/2406
2022-07-09 00:09:20 +02:00
David Ormsbee
ac3b4e3065 fix: apply LTI 1.3 XBlock patch fix 3.4.7 over nutmeg.1
The actual merge of this into open-release/nutmeg.master is here:
  https://github.com/openedx/edx-platform/pull/30716
2022-07-08 23:48:45 +02:00
David Ormsbee
6ac1c0d732 fix: replica set error from pymongo 3.10 -> 3.12 upgrade
The pymongo dependency for edx-platform was updated (3.10.1 to 3.12.3)
in https://github.com/openedx/edx-platform/pull/30569

This caused the following error when running the edx-platform database
migration split_modulestore_django.0002_data_migration as part of
`tutor dev quickstart`:

  pymongo.errors.ServerSelectionTimeoutError: client is configured to
  connect to a replica set named '' but this node belongs to a set named
  'None', Timeout: 30s, Topology Description: <TopologyDescription id:
  62bdbaf182687350acf1aeec, topology_type: Single, servers:
  [<ServerDescription ('mongodb', 27017) server_type: Unknown, rtt:
   None, error=ConfigurationError("client is configured to connect to a
   replica set named '' but this node belongs to a set named 'None'")>]>

This commit explicitly sets replicaSet to None to indicate that it's a
standalone MongoDB instance. I also had to remove the CONTENTSTORE entry
from auth.yml because edx-platform's devstack.py assumes it has a
non-null value (set in common.py), and devstack.py executes before
tutor's development.py can set this replicaSet value.
2022-07-05 16:44:12 +02:00
Overhang.IO
9d5e94b8d5 Merge remote-tracking branch 'origin/master' into nightly 2022-07-04 05:09:54 +00:00
Prakash Divy
9a63dc70ce docs: Update help command in local.rst 2022-07-04 06:30:29 +02:00
Overhang.IO
ede972a1da Merge remote-tracking branch 'origin/master' into nightly 2022-06-28 09:12:52 +00:00
Foad Lind
b8f773a5ef feat: Make it possible to customize k8s resources
Currently there is no way for plugins to customize Kubernetes resources
defined in Tutor deployment manifests.
This change makes that possible by taking advantage of the strategic
merge patching mechanism in `kustomization.yml`.
Any resource definition in a `k8s-override` patch in a plugin will
override the resource defined by Tutor, provided that their names match.

Reference: https://github.com/overhangio/tutor/pull/675
2022-06-28 10:30:24 +02:00
Overhang.IO
9d6d486c65 Merge remote-tracking branch 'origin/master' into nightly 2022-06-27 14:50:26 +00:00
Régis Behmo
4a808fa87b v14.0.2 (2022-06-27)
- [Bugfix] Update problem with hint template so it works with newer python versions. (by @mariajgrimaldi)
- [Feature] Add default PYTHONBREAKPOINT to openedx/Dockerfile (by @Carlos-Muniz)
- [Bugfix] Fix smtp server port in `cms.yml` which was causing email sending failures in the Studio. (by @regisb)
- [Bugfix] Skip waiting for MongoDB if it is served using SRV records. (by @gabor-boros)
- [Improvement] Use `git am` instead of `cherry-pick` to simplify patching process.
- [Improvement] Tutor is now compatible with Docker Compose subcommand.
2022-06-27 16:07:08 +02:00
Régis Behmo
570639a114 Merge branch 'master' into nightly 2022-06-27 10:02:34 +02:00
Gábor Boros
63bc184eab chore: remove extra empty line 2022-06-27 09:16:13 +02:00
Gabor Boros
eadb04fee9 fix: skip waiting for mongodb SRV records
Signed-off-by: Gabor Boros <gabor.brs@gmail.com>
2022-06-27 09:16:13 +02:00
Maria Grimaldi
4b8cf4ccc7 fix: update problem with hint according newer python versions 2022-06-27 09:15:56 +02:00
Overhang.IO
7f1162d29b Merge remote-tracking branch 'origin/master' into nightly 2022-06-22 10:09:09 +00:00
Régis Behmo
b13d4da1b1 fix: @lru_cache python 3.7 compatibility
In Python 3.7, which is used in the Tutor CI, the lru_cache needs an argument:
https://docs.python.org/3.7/library/functools.html#functools.lru_cache
On GitHub, CI did not catch this error because it is running Python 3.8.
2022-06-22 11:28:27 +02:00
Carlos Muniz
4dac13923f
feat: default to ipdb as PYTHONBREAKPOINT
PYTHONBREAKPOINT has been exposed as an environment variable in
the openedx Dockerfile available to be changed in config.yml. The docs have also been changed to recommend using
breakpoint and explaining how PYTHONBREAKPOINT can be modified to use a
custom debugger.

Close https://github.com/overhangio/2u-tutor-adoption/issues/45
2022-06-22 11:26:50 +02:00
iamcristye@outlook.com
a4aa01987b improvement: compatibility with Docker Compose subcommand
closes #641
2022-06-22 11:23:39 +02:00
Overhang.IO
2668633974 Merge remote-tracking branch 'origin/master' into nightly 2022-06-20 13:13:12 +00:00
Régis Behmo
602c381d6e fix: email sending from cms
Incorrect format of cms.yml config file was causing the following error on course import:

	cms-worker_1                 | Traceback (most recent call last):
	cms-worker_1                 |   File "/openedx/edx-platform/cms/djangoapps/cms_user_tasks/tasks.py", line 53, in send_task_complete_email
	cms-worker_1                 |     mail.send_mail(subject, message, from_address, [dest_addr], fail_silently=False)
	cms-worker_1                 |   File "/openedx/venv/lib/python3.8/site-packages/django/core/mail/__init__.py", line 61, in send_mail
	cms-worker_1                 |     return mail.send()
	cms-worker_1                 |   File "/openedx/venv/lib/python3.8/site-packages/django/core/mail/message.py", line 284, in send
	cms-worker_1                 |     return self.get_connection(fail_silently).send_messages([self])
	cms-worker_1                 |   File "/openedx/venv/lib/python3.8/site-packages/django/core/mail/backends/smtp.py", line 102, in send_messages
	cms-worker_1                 |     new_conn_created = self.open()
	cms-worker_1                 |   File "/openedx/venv/lib/python3.8/site-packages/django/core/mail/backends/smtp.py", line 62, in open
	cms-worker_1                 |     self.connection = self.connection_class(self.host, self.port, **connection_params)
	cms-worker_1                 |   File "/opt/pyenv/versions/3.8.12/lib/python3.8/smtplib.py", line 255, in __init__
	cms-worker_1                 |     (code, msg) = self.connect(host, port)
	cms-worker_1                 |   File "/opt/pyenv/versions/3.8.12/lib/python3.8/smtplib.py", line 339, in connect
	cms-worker_1                 |     self.sock = self._get_socket(host, port, self.timeout)
	cms-worker_1                 |   File "/opt/pyenv/versions/3.8.12/lib/python3.8/smtplib.py", line 310, in _get_socket
	cms-worker_1                 |     return socket.create_connection((host, port), timeout,
	cms-worker_1                 |   File "/opt/pyenv/versions/3.8.12/lib/python3.8/socket.py", line 787, in create_connection
	cms-worker_1                 |     for res in getaddrinfo(host, port, 0, SOCK_STREAM):
	cms-worker_1                 |   File "/opt/pyenv/versions/3.8.12/lib/python3.8/socket.py", line 918, in getaddrinfo
	cms-worker_1                 |     for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
	cms-worker_1                 | socket.gaierror: [Errno -8] Servname not supported for ai_socktype

The reason was that the trailing comma "," was interpreted as being part of the email port.
2022-06-20 14:33:48 +02:00
Régis Behmo
c014a05504 Merge branch 'master' into nightly 2022-06-16 18:04:03 +02:00
iamcristye@outlook.com
e5850eb02e improvement: use git am instead of cherry-pick 2022-06-16 17:19:54 +02:00
iamcristye@outlook.com
30e133ce07 fix: dead links 2022-06-16 17:06:57 +02:00
Overhang.IO
889725b2e0 Merge remote-tracking branch 'origin/master' into nightly 2022-06-13 20:34:46 +00:00
Régis Behmo
8a305faa6d v14.0.1 (2022-06-13)
- [Bugfix] Fix missing tables on migration from maple ([#689](https://github.com/overhangio/tutor/issues/689)). (by @ibrmora)
- [Bugfix] Fix plugin patches in cms.env.yml template.
2022-06-13 22:26:40 +02:00
ibrmora
e57e6cb2a8 fix: upgrade_from_maple issue
fix: Migration needed before upgrade_from_maple issue #689
2022-06-13 21:54:45 +02:00
Overhang.IO
4dd8e87433 Merge remote-tracking branch 'origin/master' into nightly 2022-06-11 09:14:16 +00:00
Andrés González
8b31237a7f Add missing indentation in cms feature patches 2022-06-11 10:33:50 +02:00
Régis Behmo
4af9fed338 Merge branch 'master' into nightly 2022-06-09 20:01:26 +02:00
Régis Behmo
c4388e134c v14.0.0: upgrade to Nutmeg
- 💥 [Feature] Upgrade to Nutmeg: (by @regisb)
    - 💥 [Feature] Persistent grades are now enabled by default.
    - [Bugfix] Remove edX references from bulk emails ([issue](https://github.com/openedx/build-test-release-wg/issues/100)).
    - [Improvement] For Tutor Nightly (and only Nightly), official plugins are now installed from their nightly branches on GitHub instead of a version range on PyPI. This will allow Nightly users to install all official plugins by running ``pip install -e ".[full]"``.
    - [Bugfix] Start MongoDB when running migrations, because a new data migration fails if MongoDB is not running
2022-06-09 19:18:20 +02:00
Overhang.IO
ec31b82c24 Merge remote-tracking branch 'origin/master' into nightly 2022-06-07 12:58:08 +00:00
Ghassan Maslamani
57cb956b76 fix: typo in patch name 2022-06-07 14:20:17 +02:00
Overhang.IO
7205c64e0c Merge remote-tracking branch 'origin/master' into nightly 2022-06-06 14:30:49 +00:00
Régis Behmo
698da4b1ad v13.3.1 (2022-06-06)
- [Fix] Crashing celery workers in development (#681). (by @regisb)
- [Fix] Fix studio logout issue. (by @Alec4r)
2022-06-06 16:28:47 +02:00
Régis Behmo
74212da9e2 fix: celery workers in dev
Celery workers failed to start in development with the following stacktrace:

    cms-worker_1                 | Traceback (most recent call last):
    cms-worker_1                 |   File "/openedx/venv/bin/celery", line 8, in <module>
    cms-worker_1                 |     sys.exit(main())
    cms-worker_1                 |   File "/openedx/venv/lib/python3.8/site-packages/celery/__main__.py", line 16, in main
    cms-worker_1                 |     _main()
    cms-worker_1                 |   File "/openedx/venv/lib/python3.8/site-packages/celery/bin/celery.py", line 322, in main
    cms-worker_1                 |     cmd.execute_from_commandline(argv)
    cms-worker_1                 |   File "/openedx/venv/lib/python3.8/site-packages/celery/bin/celery.py", line 499, in execute_from_commandline
    cms-worker_1                 |     super(CeleryCommand, self).execute_from_commandline(argv)))
    cms-worker_1                 |   File "/openedx/venv/lib/python3.8/site-packages/celery/bin/base.py", line 305, in execute_from_commandline
    cms-worker_1                 |     return self.handle_argv(self.prog_name, argv[1:])
    cms-worker_1                 |   File "/openedx/venv/lib/python3.8/site-packages/celery/bin/celery.py", line 491, in handle_argv
    cms-worker_1                 |     return self.execute(command, argv)
    cms-worker_1                 |   File "/openedx/venv/lib/python3.8/site-packages/celery/bin/celery.py", line 415, in execute
    cms-worker_1                 |     return cls(
    cms-worker_1                 |   File "/openedx/venv/lib/python3.8/site-packages/celery/bin/worker.py", line 221, in run_from_argv
    cms-worker_1                 |     *self.parse_options(prog_name, argv, command))
    cms-worker_1                 |   File "/openedx/venv/lib/python3.8/site-packages/celery/bin/base.py", line 428, in parse_options
    cms-worker_1                 |     self.parser = self.create_parser(prog_name, command)
    cms-worker_1                 |   File "/openedx/venv/lib/python3.8/site-packages/celery/bin/base.py", line 440, in create_parser
    cms-worker_1                 |     description=self._format_description(self.description),
    cms-worker_1                 |   File "/openedx/venv/lib/python3.8/site-packages/celery/bin/base.py", line 462, in _format_description
    cms-worker_1                 |     text.fill_paragraphs(text.dedent(description), width))
    cms-worker_1                 |   File "/openedx/venv/lib/python3.8/site-packages/celery/utils/text.py", line 58, in fill_paragraphs
    cms-worker_1                 |     return sep.join(fill(p, width) for p in s.split(sep))
    cms-worker_1                 |   File "/openedx/venv/lib/python3.8/site-packages/celery/utils/text.py", line 58, in <genexpr>
    cms-worker_1                 |     return sep.join(fill(p, width) for p in s.split(sep))
    cms-worker_1                 |   File "/opt/pyenv/versions/3.8.12/lib/python3.8/textwrap.py", line 391, in fill
    cms-worker_1                 |     return w.fill(text)
    cms-worker_1                 |   File "/opt/pyenv/versions/3.8.12/lib/python3.8/textwrap.py", line 363, in fill
    cms-worker_1                 |     return "\n".join(self.wrap(text))
    cms-worker_1                 |   File "/opt/pyenv/versions/3.8.12/lib/python3.8/textwrap.py", line 354, in wrap
    cms-worker_1                 |     return self._wrap_chunks(chunks)
    cms-worker_1                 |   File "/opt/pyenv/versions/3.8.12/lib/python3.8/textwrap.py", line 248, in _wrap_chunks
    cms-worker_1                 |     raise ValueError("invalid width %r (must be > 0)" % self.width)
    cms-worker_1                 | ValueError: invalid width -2 (must be > 0)

This issue was reported upstream here: https://github.com/celery/celery/issues/6302

It is caused by the `tty: true` statement, for some reason. It will be fixed in
Nutmeg, after celery is upgraded to 5.2.6.

Close #681.
2022-06-06 16:25:35 +02:00