christianlight
/
tutor
mirror of https://github.com/ChristianLight/tutor.git
7
0
Fork 0
Code Issues Releases Activity
tutor/Makefile

124 lines
4.1 KiB
Makefile
Raw Normal View History

Add a self-documenting Make help Picked from https://github.com/openfun/fun-platform, which itself picked it from https://marmelab.com/blog/2016/02/29/auto-documented-makefile.html Running `make help` or simply `make` now produces a beautiful colored help.
2018-09-29 15:08:39 +00:00
.DEFAULT_GOAL := help
Add "docs" target to root makefile
2020-01-10 14:30:09 +00:00
.PHONY: docs
refactor: add code coverage, cover CLI commands with tests
2021-11-23 08:25:09 +00:00
SRC_DIRS = ./tutor ./tests ./bin ./docs
Working Kubernetes quickstart The k8s quickstart command is now functional, with suppport for https, xqueue, notes and minio. There are still a few bugs to get rid of, though.
2019-06-06 19:58:21 +00:00
BLACK_OPTS = --exclude templates ${SRC_DIRS}
Add a self-documenting Make help Picked from https://github.com/openfun/fun-platform, which itself picked it from https://marmelab.com/blog/2016/02/29/auto-documented-makefile.html Running `make help` or simply `make` now produces a beautiful colored help.
2018-09-29 15:08:39 +00:00
Simplify environment generation Environment is no longer generated separately for each target, but only once the configuration is saved. Note that the environment is automatically updated during re-configuration, based on a "version" file stored in the environment.
2019-03-18 16:26:37 +00:00
###### Development
build: 'make help' works nicer on Mac sed on Mac doesn't seem to understand \n. While I was in here, I fixed some typos.
2022-01-05 15:46:19 +00:00
docs: ## Build HTML documentation
Add "docs" target to root makefile
2020-01-10 14:30:09 +00:00
$(MAKE) -C docs
Tutor v3 complete rewrite Replace all make commands by a single "tutor" binary. Environment and data are all moved to ~/.tutor/local/share/tutor. We take the opportunity to add a web UI and revamp the documentation. This is a complete rewrite. Close #121. Close #147.
2019-01-22 20:25:04 +00:00
compile-requirements: ## Compile requirements files
Upgrade all requirements Also, add a `upgrade-requirements` target.
2020-01-10 14:38:17 +00:00
pip-compile requirements/base.in
pip-compile requirements/dev.in
pip-compile requirements/docs.in
upgrade-requirements: ## Upgrade requirements files
pip-compile --upgrade requirements/base.in
pip-compile --upgrade requirements/dev.in
pip-compile --upgrade requirements/docs.in
Push to repo as part of "make release"
2020-06-21 11:47:14 +00:00
build: 'make help' works nicer on Mac sed on Mac doesn't seem to understand \n. While I was in here, I fixed some typos.
2022-01-05 15:46:19 +00:00
build-pythonpackage: build-pythonpackage-tutor ## Build Python packages ready to upload to pypi
fix: make sure that tutor-openedx is an empty package Previously, the tutor-openedx package was loading tons of template data from the MANIFEST.in. Turns out, we cannot ignore the MANIFEST.in file with setuptools. So we need to move tutor-openedx to a separate, dedicated folder. To auto-discover the package version, we copy it at runtime (in the make command).
2021-07-06 07:42:57 +00:00
build-pythonpackage-tutor: ## Build the "tutor" python package for upload to pypi
Simplify CI This comes after a big refactoring of our CI pipeline.
2020-10-02 11:34:54 +00:00
python setup.py sdist
fix: make sure that tutor-openedx is an empty package Previously, the tutor-openedx package was loading tons of template data from the MANIFEST.in. Turns out, we cannot ignore the MANIFEST.in file with setuptools. So we need to move tutor-openedx to a separate, dedicated folder. To auto-discover the package version, we copy it at runtime (in the make command).
2021-07-06 07:42:57 +00:00
Clean some minor stuff from Makefile
2021-03-17 17:19:43 +00:00
push-pythonpackage: ## Push python package to pypi
feat: switch the package name from "tutor-openedx" to "tutor" The package maintainer of the "tutor" package was kind enough to transfer ownership of the project to us. This is great, because we no longer have to use the "openedx" suffix, which is trademarked. For the time being, we keep maintaining the "tutor-openedx" package which has a 1-to-1 dependency on the "tutor" package. In the future, we expect that we will no longer push upgrades to tutor-openedx.
2021-06-25 15:03:43 +00:00
twine upload --skip-existing dist/tutor-$(shell make version).tar.gz
Simplify CI This comes after a big refactoring of our CI pipeline.
2020-10-02 11:34:54 +00:00
build: 'make help' works nicer on Mac sed on Mac doesn't seem to understand \n. While I was in here, I fixed some typos.
2022-01-05 15:46:19 +00:00
test: test-lint test-unit test-types test-format test-pythonpackage ## Run all tests by decreasing order of priority
Introduce automatic code formatting/linting Code formatting makes sure that the python code looks decent, but it does not check for coding errors. https://black.readthedocs.io/en/stable/ Code linting runs static error detection on the python code, but does not bother about formatting: https://pylint.readthedocs.io/en/latest/
2019-05-05 09:40:49 +00:00
feat: migrate to plugins.v1 with filters & actions This is a very large refactoring which aims at making Tutor both more extendable and more generic. Historically, the Tutor plugin system was designed as an ad-hoc solution to allow developers to modify their own Open edX platforms without having to fork Tutor. The plugin API was simple, but limited, because of its ad-hoc nature. As a consequence, there were many things that plugin developers could not do, such as extending different parts of the CLI or adding custom template filters. Here, we refactor the whole codebase to make use of a generic plugin system. This system was inspired by the Wordpress plugin API and the Open edX "hooks and filters" API. The various components are added to a small core thanks to a set of actions and filters. Actions are callback functions that can be triggered at different points of the application lifecycle. Filters are functions that modify some data. Both actions and filters are collectively named as "hooks". Hooks can optionally be created within a certain context, which makes it easier to keep track of which application created which callback. This new hooks system allows us to provide a Python API that developers can use to extend their applications. The API reference is added to the documentation, along with a new plugin development tutorial. The plugin v0 API remains supported for backward compatibility of existing plugins. Done: - Do not load commands from plugins which are not enabled. - Load enabled plugins once on start. - Implement contexts for actions and filters, which allow us to keep track of the source of every hook. - Migrate patches - Migrate commands - Migrate plugin detection - Migrate templates_root - Migrate config - Migrate template environment globals and filters - Migrate hooks to tasks - Generate hook documentation - Generate patch reference documentation - Add the concept of action priority Close #499.
2022-02-07 17:11:43 +00:00
test-static: test-lint test-types test-format ## Run only static tests
Introduce automatic code formatting/linting Code formatting makes sure that the python code looks decent, but it does not check for coding errors. https://black.readthedocs.io/en/stable/ Code linting runs static error detection on the python code, but does not bother about formatting: https://pylint.readthedocs.io/en/latest/
2019-05-05 09:40:49 +00:00
test-format: ## Run code formatting tests
No automated formatting for templates
2019-06-03 10:23:05 +00:00
black --check --diff $(BLACK_OPTS)
Introduce automatic code formatting/linting Code formatting makes sure that the python code looks decent, but it does not check for coding errors. https://black.readthedocs.io/en/stable/ Code linting runs static error detection on the python code, but does not bother about formatting: https://pylint.readthedocs.io/en/latest/
2019-05-05 09:40:49 +00:00
test-lint: ## Run code linting tests
feat: migrate to plugins.v1 with filters & actions This is a very large refactoring which aims at making Tutor both more extendable and more generic. Historically, the Tutor plugin system was designed as an ad-hoc solution to allow developers to modify their own Open edX platforms without having to fork Tutor. The plugin API was simple, but limited, because of its ad-hoc nature. As a consequence, there were many things that plugin developers could not do, such as extending different parts of the CLI or adding custom template filters. Here, we refactor the whole codebase to make use of a generic plugin system. This system was inspired by the Wordpress plugin API and the Open edX "hooks and filters" API. The various components are added to a small core thanks to a set of actions and filters. Actions are callback functions that can be triggered at different points of the application lifecycle. Filters are functions that modify some data. Both actions and filters are collectively named as "hooks". Hooks can optionally be created within a certain context, which makes it easier to keep track of which application created which callback. This new hooks system allows us to provide a Python API that developers can use to extend their applications. The API reference is added to the documentation, along with a new plugin development tutorial. The plugin v0 API remains supported for backward compatibility of existing plugins. Done: - Do not load commands from plugins which are not enabled. - Load enabled plugins once on start. - Implement contexts for actions and filters, which allow us to keep track of the source of every hook. - Migrate patches - Migrate commands - Migrate plugin detection - Migrate templates_root - Migrate config - Migrate template environment globals and filters - Migrate hooks to tasks - Generate hook documentation - Generate patch reference documentation - Add the concept of action priority Close #499.
2022-02-07 17:11:43 +00:00
pylint --errors-only --enable=unused-import,unused-argument --ignore=templates --ignore=docs/_ext ${SRC_DIRS}
Introduce automatic code formatting/linting Code formatting makes sure that the python code looks decent, but it does not check for coding errors. https://black.readthedocs.io/en/stable/ Code linting runs static error detection on the python code, but does not bother about formatting: https://pylint.readthedocs.io/en/latest/
2019-05-05 09:40:49 +00:00
Move plugins to dedicated repositories minio, notes and xqueue plugins are moved to tutor-* github repos.
2019-08-21 16:32:39 +00:00
test-unit: ## Run unit tests
Simplify CI This comes after a big refactoring of our CI pipeline.
2020-10-02 11:34:54 +00:00
python -m unittest discover tests
Add unit tests! yay! Now, that was seriously missing.
2019-03-23 23:16:03 +00:00
refactor: add type annotations Annotations were generated with pyannotate: https://github.com/dropbox/pyannotate We are running in strict mode, which is awesome! This affects a large part of the code base, which might be an issue for people running a fork of Tutor. Nonetheless, the behavior should not be affected. If anything, this process has helped find and resolve a few type-related bugs. Thus, this is not considered as a breaking change.
2021-02-25 08:09:14 +00:00
test-types: ## Check type definitions
Merge branch 'master' into nightly
2022-11-15 15:59:19 +00:00
mypy --exclude=templates --ignore-missing-imports --implicit-reexport --strict ${SRC_DIRS}
refactor: add type annotations Annotations were generated with pyannotate: https://github.com/dropbox/pyannotate We are running in strict mode, which is awesome! This affects a large part of the code base, which might be an issue for people running a fork of Tutor. Nonetheless, the behavior should not be affected. If anything, this process has helped find and resolve a few type-related bugs. Thus, this is not considered as a breaking change.
2021-02-25 08:09:14 +00:00
Simplify CI This comes after a big refactoring of our CI pipeline.
2020-10-02 11:34:54 +00:00
test-pythonpackage: build-pythonpackage ## Test that package can be uploaded to pypi
feat: switch the package name from "tutor-openedx" to "tutor" The package maintainer of the "tutor" package was kind enough to transfer ownership of the project to us. This is great, because we no longer have to use the "openedx" suffix, which is trademarked. For the time being, we keep maintaining the "tutor-openedx" package which has a 1-to-1 dependency on the "tutor" package. In the future, we expect that we will no longer push upgrades to tutor-openedx.
2021-06-25 15:03:43 +00:00
twine check dist/tutor-$(shell make version).tar.gz
Test packaging as part of full test suite We've had issues before with improperly formatted README.rst which was causing pypi upload failure.
2019-06-19 10:02:28 +00:00
feat: add a convenient `tutor k8s apply` command This is convenient to check k8s template validity.
2022-02-21 10:53:17 +00:00
test-k8s: ## Validate the k8s format with kubectl. Not part of the standard test suite.
tutor k8s apply --dry-run=client --validate=true
Introduce automatic code formatting/linting Code formatting makes sure that the python code looks decent, but it does not check for coding errors. https://black.readthedocs.io/en/stable/ Code linting runs static error detection on the python code, but does not bother about formatting: https://pylint.readthedocs.io/en/latest/
2019-05-05 09:40:49 +00:00
format: ## Format code automatically
No automated formatting for templates
2019-06-03 10:23:05 +00:00
black $(BLACK_OPTS)
Introduce automatic code formatting/linting Code formatting makes sure that the python code looks decent, but it does not check for coding errors. https://black.readthedocs.io/en/stable/ Code linting runs static error detection on the python code, but does not bother about formatting: https://pylint.readthedocs.io/en/latest/
2019-05-05 09:40:49 +00:00
feat: migrate to plugins.v1 with filters & actions This is a very large refactoring which aims at making Tutor both more extendable and more generic. Historically, the Tutor plugin system was designed as an ad-hoc solution to allow developers to modify their own Open edX platforms without having to fork Tutor. The plugin API was simple, but limited, because of its ad-hoc nature. As a consequence, there were many things that plugin developers could not do, such as extending different parts of the CLI or adding custom template filters. Here, we refactor the whole codebase to make use of a generic plugin system. This system was inspired by the Wordpress plugin API and the Open edX "hooks and filters" API. The various components are added to a small core thanks to a set of actions and filters. Actions are callback functions that can be triggered at different points of the application lifecycle. Filters are functions that modify some data. Both actions and filters are collectively named as "hooks". Hooks can optionally be created within a certain context, which makes it easier to keep track of which application created which callback. This new hooks system allows us to provide a Python API that developers can use to extend their applications. The API reference is added to the documentation, along with a new plugin development tutorial. The plugin v0 API remains supported for backward compatibility of existing plugins. Done: - Do not load commands from plugins which are not enabled. - Load enabled plugins once on start. - Implement contexts for actions and filters, which allow us to keep track of the source of every hook. - Migrate patches - Migrate commands - Migrate plugin detection - Migrate templates_root - Migrate config - Migrate template environment globals and filters - Migrate hooks to tasks - Generate hook documentation - Generate patch reference documentation - Add the concept of action priority Close #499.
2022-02-07 17:11:43 +00:00
isort: ## Sort imports. This target is not mandatory because the output may be incompatible with black formatting. Provided for convenience purposes.
isort --skip=templates ${SRC_DIRS}
ci: better github release management with scriv In scriv 1.1.0 the GitHub release description can be templated: https://github.com/nedbat/scriv/issues/61 https://github.com/nedbat/scriv/releases/tag/1.1.0 This means that we can finally get rid of our ugly scripts to generate the release description \o/
2023-01-17 10:58:39 +00:00
changelog-entry: ## Create a new changelog entry
scriv create
Simplify CI This comes after a big refactoring of our CI pipeline.
2020-10-02 11:34:54 +00:00
ci: better github release management with scriv In scriv 1.1.0 the GitHub release description can be templated: https://github.com/nedbat/scriv/issues/61 https://github.com/nedbat/scriv/releases/tag/1.1.0 This means that we can finally get rid of our ugly scripts to generate the release description \o/
2023-01-17 10:58:39 +00:00
changelog: ## Collect changelog entries in the CHANGELOG.md file
scriv collect
Simplify CI This comes after a big refactoring of our CI pipeline.
2020-10-02 11:34:54 +00:00
refactor: add code coverage, cover CLI commands with tests
2021-11-23 08:25:09 +00:00
###### Code coverage
coverage: ## Run unit-tests before analyzing code coverage and generate report
$(MAKE) --keep-going coverage-tests coverage-report
coverage-tests: ## Run unit-tests and analyze code coverage
coverage run -m unittest discover
coverage-report: ## Generate CLI report for the code coverage
coverage report
coverage-html: coverage-report ## Generate HTML report for the code coverage
coverage html
coverage-browse-report: coverage-html ## Open the HTML report in the browser
sensible-browser htmlcov/index.html
ci: better github release management with scriv In scriv 1.1.0 the GitHub release description can be templated: https://github.com/nedbat/scriv/issues/61 https://github.com/nedbat/scriv/releases/tag/1.1.0 This means that we can finally get rid of our ugly scripts to generate the release description \o/
2023-01-17 10:58:39 +00:00
###### Continuous integration tasks
Simplify environment generation Environment is no longer generated separately for each target, but only once the configuration is saved. Note that the environment is automatically updated during re-configuration, based on a "version" file stored in the environment.
2019-03-18 16:26:37 +00:00
Tutor v3 complete rewrite Replace all make commands by a single "tutor" binary. Environment and data are all moved to ~/.tutor/local/share/tutor. We take the opportunity to add a web UI and revamp the documentation. This is a complete rewrite. Close #121. Close #147.
2019-01-22 20:25:04 +00:00
bundle: ## Bundle the tutor package in a single "dist/tutor" executable
Add all plugins (with data) to binary bundle All existing plugins are added to the binary bundle, in their latest version, so that users don't need to pip install tutor. Also, the tutor MANIFEST.in file was removed to simplify the management of package data. Close #242.
2019-08-20 15:01:34 +00:00
pyinstaller tutor.spec
Add `make version` command This makes tagging simpler.
2019-02-24 19:14:25 +00:00
ci: better github release management with scriv In scriv 1.1.0 the GitHub release description can be templated: https://github.com/nedbat/scriv/issues/61 https://github.com/nedbat/scriv/releases/tag/1.1.0 This means that we can finally get rid of our ugly scripts to generate the release description \o/
2023-01-17 10:58:39 +00:00
bootstrap-dev: ## Install dev requirements
pip install .
pip install -r requirements/dev.txt
docs: migrate to scriv to manage changelog Changelog management was starting to be a hassle: - there were conflicts every time a PR was merged - there were conflicts every time we merged the nightly branch in the new release branch, or vice versa. Now, all changelog entries are stored as separate files in changelog.d, including nightly. Nightly entries will be collected for every major release.
2022-11-21 15:35:14 +00:00
ci: better github release management with scriv In scriv 1.1.0 the GitHub release description can be templated: https://github.com/nedbat/scriv/issues/61 https://github.com/nedbat/scriv/releases/tag/1.1.0 This means that we can finally get rid of our ugly scripts to generate the release description \o/
2023-01-17 10:58:39 +00:00
bootstrap-dev-plugins: bootstrap-dev ## Install dev requirements and all supported plugins
pip install -r requirements/plugins.txt
Simplify environment generation Environment is no longer generated separately for each target, but only once the configuration is saved. Note that the environment is automatically updated during re-configuration, based on a "version" file stored in the environment.
2019-03-18 16:26:37 +00:00
Actually remove the `images build --pull option` This option is mostly useless to us in CI, as it attempts (and fails) to update the base image when building the dev image. For good reason: the base image of the dev image is the latest openedx, which has not been pushed to docker.io at the time of building -_-
2021-03-17 17:19:27 +00:00
pull-base-images: # Manually pull base images
docker image pull docker.io/ubuntu:20.04
Better CI
2019-02-10 20:02:30 +00:00
ci-info: ## Print info about environment
Simplify CI This comes after a big refactoring of our CI pipeline.
2020-10-02 11:34:54 +00:00
python --version
pip --version
Push plugin packages to pypi
2019-06-23 20:46:41 +00:00
Simplify CI This comes after a big refactoring of our CI pipeline.
2020-10-02 11:34:54 +00:00
ci-test-bundle: ## Run basic tests on bundle
Add CI command to view size of bundled tutor binary
2020-07-01 07:21:45 +00:00
ls -lh ./dist/tutor
Simplify environment generation Environment is no longer generated separately for each target, but only once the configuration is saved. Note that the environment is automatically updated during re-configuration, based on a "version" file stored in the environment.
2019-03-18 16:26:37 +00:00
./dist/tutor --version
./dist/tutor config printroot
Make configuration silent (non-interactive) by default Thus, we remove the -y/--yes options, which were kind of unintuitive, and we add instead `-i/--interactive`. The quickstart commands remain interactive by default, but can be silenced with `-I/--non-interactive`.
2019-06-05 17:45:22 +00:00
yes "" | ./dist/tutor config save --interactive
Migrate notes to a dedicated plugin
2019-07-03 14:09:33 +00:00
./dist/tutor config save
Refactor plugin internals This is for supporting json-based plugins. The great thing about this change is that it allows us to easily print plugin version numbers in `plugins list`.
2020-01-14 12:32:45 +00:00
./dist/tutor plugins list
depr: richie plugin is no longer official The richie plugin was transferred to the openfun GitHub org. As a consequence, it is no longer part of the list of official plugins. See: https://github.com/overhangio/tutor-contrib-richie/pull/5
2022-09-08 07:40:25 +00:00
./dist/tutor plugins enable android discovery ecommerce forum license mfe minio notes webui xqueue
Refactor plugin internals This is for supporting json-based plugins. The great thing about this change is that it allows us to easily print plugin version numbers in `plugins list`.
2020-01-14 12:32:45 +00:00
./dist/tutor plugins list
v11.0.0 (2020-12-09) - 💥[Improvement] Upgrade Open edX to Koa - 💥 Setting changes: - The ``ACTIVATE_HTTPS`` setting was renamed to ``ENABLE_HTTPS``. - Other ``ACTIVATE_*`` variables were all renamed to ``RUN_*``. - The ``WEB_PROXY`` setting was removed and ``RUN_CADDY`` was added. - The ``NGINX_HTTPS_PORT`` setting is deprecated. - Architectural changes: - Use Caddy as a web proxy for automated SSL/TLS certificate generation: - Nginx no longer listens to port 443 for https traffic - The Caddy configuration file comes with a new ``caddyfile`` patch for much simpler SSL/TLS management. - Configuration files for web proxies are no longer provided. - Kubernetes deployment no longer requires setting up a custom Ingress resource or custom manager. - Gunicorn and Whitenoise are replaced by uwsgi: this increases boostrap performance and makes it no longer necessary to mount media folders in the Nginx container. - Replace memcached and rabbitmq by redis. - Additional features: - Make it possible to disable all plugins at once with ``plugins disable all``. - Add ``tutor k8s wait`` command to wait for a pod to become ready - Faster, more reliable static assets with local memory caching - Deprecation: proxy files for Apache and Nginx are no longer provided out of the box. - Removed plugin `{{ patch (...) }}` statements: - "https-create", "k8s-ingress-rules", "k8s-ingress-tls-hosts": these are no longer necessary. Instead, declare your app in the "caddyfile" patch. - "local-docker-compose-nginx-volumes": this patch was primarily used to serve media assets. The recommended is now to serve assets with uwsgi.
2020-09-17 10:53:14 +00:00
./dist/tutor license --help
Better CI
2019-02-10 20:02:30 +00:00
Fix building xqueue/notes as part of CI
2019-07-04 14:28:45 +00:00
ci-bootstrap-images:
Simplify CI This comes after a big refactoring of our CI pipeline.
2020-10-02 11:34:54 +00:00
pip install .
Fix CI image building
2019-07-04 09:26:10 +00:00
tutor config save
Better release descriptions By using `github-releases`, we can now generate much better release description blurbs.
2019-04-22 13:14:46 +00:00
Simplify environment generation Environment is no longer generated separately for each target, but only once the configuration is saved. Note that the environment is automatically updated during re-configuration, based on a "version" file stored in the environment.
2019-03-18 16:26:37 +00:00
###### Additional commands
version: ## Print the current tutor version
fix: package version in nightly The string returned by "make version" should not include the "-nightly" suffix, otherwise tests fail.
2022-03-31 13:36:30 +00:00
@python -c 'import io, os; about = {}; exec(io.open(os.path.join("tutor", "__about__.py"), "rt", encoding="utf-8").read(), about); print(about["__package_version__"])'
Simplify environment generation Environment is no longer generated separately for each target, but only once the configuration is saved. Note that the environment is automatically updated during re-configuration, based on a "version" file stored in the environment.
2019-03-18 16:26:37 +00:00
Provide better names for help sections
2018-09-29 15:22:55 +00:00
ESCAPE = 
help: ## Print this help
@grep -E '^([a-zA-Z_-]+:.*?## .*|######* .+)$$' Makefile \
build: 'make help' works nicer on Mac sed on Mac doesn't seem to understand \n. While I was in here, I fixed some typos.
2022-01-05 15:46:19 +00:00
| sed 's/######* \(.*\)/@ $(ESCAPE)[1;31m\1$(ESCAPE)[0m/g' | tr '@' '\n' \
Provide better names for help sections
2018-09-29 15:22:55 +00:00
| awk 'BEGIN {FS = ":.*?## "}; {printf "\033[33m%-30s\033[0m %s\n", $$1, $$2}'