Improve lms/cms logging

- deduplicate stdout logs
- don't persist celery logs
- persist logs to all.log by default

CHANGELOG.md

@@ -2,6 +2,7 @@
 
 ## Latest
 
+- [Improvement] Persist LMS/CMS logs to disk by default (with collaboration from @silviot 💪)
 - [Bugfix] Fix installing a locally cloned requirement repository
 - [Improvement] Add `--no-cache` option to `images build`
 - [Improvement] Make it possible to configure the notes service hostname
@@ -11,7 +12,6 @@
 - [Bugfix] Fix boolean configuration choices
 
 ## 3.3.9 (2019-05-13)
-
 - [Improvement] Add `local exec` command for running commands inside existing containers
 - [Bugfix] Fix triple display of courses in LMS search (upstream patch, #189)
 

docs/local.rst

@@ -72,6 +72,22 @@ In production, you will probably want to daemonize the services. To do so, run::
 And then, to stop all services::
 
     tutor local stop
+    
+Logging
+~~~~~~~
+
+By default, logs from all containers are forwarded to the `default Docker logging driver <https://docs.docker.com/config/containers/logging/configure/>`_: this means that logs are printed to the standard output when running in non-daemon mode (``tutor local start``). In daemon mode, logs can still be accessed with ``tutor local logs`` commands (see :ref:`logging <logging>`).
+
+In addition, all LMS and CMS logs are persisted to disk by default in the following files::
+    
+    $(tutor config printroot)/data/lms/logs/all.log
+    $(tutor config printroot)/data/cms/logs/all.log
+
+Finally, tracking logs that store `user events <https://edx.readthedocs.io/projects/devdata/en/latest/internal_data_formats/tracking_logs/index.html>`_ are persisted in the following files::
+
+    $(tutor config printroot)/data/lms/logs/tracking.log
+    $(tutor config printroot)/data/cms/logs/tracking.log
+
 
 Extra commands
 --------------
@@ -188,27 +204,6 @@ The default settings module loaded by ``edx-platform`` is ``tutor.production``.
 Of course, your settings should be compatible with the docker install. You can get some inspiration from the ``production.py`` settings modules generated by Tutor, or just import it as a base by adding ``from .production import *`` at the top of ``mysettings.py``.
 
 
-Saving logs to files instead of sending them to Docker
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-By default tracking logs and app logs are sent to docker, and are visible with ``tutor local logs``. Another option is to persist logs in a directory alongside the database data.
-To direct logs from an openedx component (lms, cms or workers) edit the docker-compose file at ``$(tutor config printroot)/env/local/docker-compose.yml`` and add the followinf line to the lms/cms and worker containers, replaceing SERVICE_NAME with the relevant service name (lms, cms, lms-worker, cms-worker):
-
-    volumes:
-      - ../apps/openedx/settings/lms/:/openedx/edx-platform/lms/envs/tutor/
-      - ../apps/openedx/settings/cms/:/openedx/edx-platform/cms/envs/tutor/
-      ...
-      - ../../data/logs/SERVICE_NAME:/openedx/logs # <-- Add this line
-
-If you also want logs from nginx to be saved to disk instead of sent to docker, add the following to the nginx container in ``$(tutor config printroot)/env/local/docker-compose.yml``:
-
-    volumes:
-      - ../apps/nginx:/etc/nginx/conf.d/
-      - ../../data/openedx:/var/www/openedx:ro
-      ...
-      - ../../data/logs/lms:/var/log/nginx # <-- Add this line
-
-
 Upgrading from earlier versions
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 

tutor/templates/apps/openedx/settings/cms/common.py

@@ -11,16 +11,16 @@ VIDEO_TRANSCRIPTS_SETTINGS["STORAGE_KWARGS"]["location"] = MEDIA_ROOT
 # Change syslog-based loggers which don't work inside docker containers
 LOGGING["handlers"]["local"] = {
     "class": "logging.handlers.WatchedFileHandler",
-    "filename": "/openedx/logs/edx.log",
+    "filename": os.path.join(LOG_DIR, "all.log"),
     "formatter": "standard",
 }
-
 LOGGING["handlers"]["tracking"] = {
     "level": "DEBUG",
     "class": "logging.handlers.WatchedFileHandler",
-    "filename": "/openedx/logs/tracking.log",
+    "filename": os.path.join(LOG_DIR, "tracking.log"),
     "formatter": "standard",
 }
+LOGGING["loggers"]["tracking"]["handlers"] = ["console", "local", "tracking"]
 
 LOCALE_PATHS.append("/openedx/locale")
 

tutor/templates/apps/openedx/settings/lms/common.py

@@ -13,16 +13,16 @@ VIDEO_TRANSCRIPTS_SETTINGS["STORAGE_KWARGS"]["location"] = MEDIA_ROOT
 # Change syslog-based loggers which don't work inside docker containers
 LOGGING["handlers"]["local"] = {
     "class": "logging.handlers.WatchedFileHandler",
-    "filename": "/openedx/logs/edx.log",
+    "filename": os.path.join(LOG_DIR, "all.log"),
     "formatter": "standard",
 }
-
 LOGGING["handlers"]["tracking"] = {
     "level": "DEBUG",
     "class": "logging.handlers.WatchedFileHandler",
-    "filename": "/openedx/logs/tracking.log",
+    "filename": os.path.join(LOG_DIR, "tracking.log"),
     "formatter": "standard",
 }
+LOGGING["loggers"]["tracking"]["handlers"] = ["console", "local", "tracking"]
 
 # Fix media files paths
 VIDEO_IMAGE_SETTINGS["STORAGE_KWARGS"]["location"] = MEDIA_ROOT

tutor/templates/build/openedx/Dockerfile

@@ -93,11 +93,6 @@ ENV SETTINGS tutor.production
 # Entrypoint will fix permissions of all files and run commands as openedx
 ENTRYPOINT ["docker-entrypoint.sh"]
 
-# Configure logging
-RUN mkdir -p /openedx/logs \
-    && ln -s /dev/stdout /openedx/logs/tracking.log \
-    && ln -s /dev/stderr /openedx/logs/edx.log
-
 # Run server
 COPY gunicorn_conf.py /openedx/gunicorn_conf.py
 EXPOSE 8000

tutor/templates/local/docker-compose.yml

@@ -150,7 +150,7 @@ services:
       SERVICE_VARIANT: lms
       SETTINGS: ${EDX_PLATFORM_SETTINGS:-tutor.production}
       C_FORCE_ROOT: "1" # run celery tasks as root #nofear
-    command: ./manage.py lms celery worker --logfile=/openedx/logs/edx.log --loglevel=info --hostname=edx.lms.core.default.%%h --maxtasksperchild 100
+    command: ./manage.py lms celery worker --loglevel=info --hostname=edx.lms.core.default.%%h --maxtasksperchild 100
     restart: unless-stopped
     volumes:
       - ../apps/openedx/settings/lms/:/openedx/edx-platform/lms/envs/tutor/
@@ -168,7 +168,7 @@ services:
       SERVICE_VARIANT: cms
       SETTINGS: ${EDX_PLATFORM_SETTINGS:-tutor.production}
       C_FORCE_ROOT: "1" # run celery tasks as root #nofear
-    command: ./manage.py cms celery worker --logfile=/openedx/logs/edx.log --loglevel=info --hostname=edx.cms.core.default.%%h --maxtasksperchild 100
+    command: ./manage.py cms celery worker --loglevel=info --hostname=edx.cms.core.default.%%h --maxtasksperchild 100
     restart: unless-stopped
     volumes:
       - ../apps/openedx/settings/lms/:/openedx/edx-platform/lms/envs/tutor/