accounting/bench
2
0
Fork 0
mirror of https://github.com/frappe/bench.git synced 2024-11-11 15:51:03 +00:00
Code Issues Releases Activity

fix: Prevent Gunicorn child workers hangup

Browse Source 
Problem:

- Prerequisite: long running reqeust is going on.
- `bench restart` is requested.
- supervisord sends sigterm to gunicorn master process.
- gunicorn passes it to all child table and waits for graceful shutdown (30 seconds)
- supervisord has no chill, and sends sigkill to master process in 10 seconds
- gunicorn master proesss is dead, so now child workers will keep running until they complete request which can take a really long time.
- This entire time the sites will be down.

Fix:
- Explicitly encode default graceful_timeout in config - 30 seconds.
- Make supervisor wait 10 more seconds for gunicorn to do its thing, then only send sigkill.
This commit is contained in:
Ankush Menat 2023-11-02 11:40:07 +05:30
parent 53a8fedadc
commit 36f2194865
1 changed files with 3 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
bench/config/templates/supervisor.conf
View File

@ -2,13 +2,15 @@
; priority=1 --> Lower priorities indicate programs that start first and shut down last ; priority=1 --> Lower priorities indicate programs that start first and shut down last
; killasgroup=true --> send kill signal to child processes too ; killasgroup=true --> send kill signal to child processes too
; graceful timeout should always be lower than stopwaitsecs to avoid orphan gunicorn workers.
[program:{{ bench_name }}-frappe-web] [program:{{ bench_name }}-frappe-web]
command={{ bench_dir }}/env/bin/gunicorn -b 127.0.0.1:{{ webserver_port }} -w {{ gunicorn_workers }} --max-requests {{ gunicorn_max_requests }} --max-requests-jitter {{ gunicorn_max_requests_jitter }} -t {{ http_timeout }} frappe.app:application --preload command={{ bench_dir }}/env/bin/gunicorn -b 127.0.0.1:{{ webserver_port }} -w {{ gunicorn_workers }} --max-requests {{ gunicorn_max_requests }} --max-requests-jitter {{ gunicorn_max_requests_jitter }} -t {{ http_timeout }} --graceful-timeout 30 frappe.app:application --preload
priority=4 priority=4
autostart=true autostart=true
autorestart=true autorestart=true
stdout_logfile={{ bench_dir }}/logs/web.log stdout_logfile={{ bench_dir }}/logs/web.log
stderr_logfile={{ bench_dir }}/logs/web.error.log stderr_logfile={{ bench_dir }}/logs/web.error.log
stopwaitsecs=40
user={{ user }} user={{ user }}
directory={{ sites_dir }} directory={{ sites_dir }}