From 09274ed3050ebf219fb643889569fbbce40fbaec Mon Sep 17 00:00:00 2001 From: Ankush Menat Date: Thu, 20 Oct 2022 16:09:06 +0530 Subject: [PATCH 1/9] docs: remove easy install from docs (#1380) https://discuss.erpnext.com/t/deprecation-easy-install-script-is-no-longer-supported/96245/3 [skip ci] --- README.md | 74 ++----------------------------------------------------- 1 file changed, 2 insertions(+), 72 deletions(-) diff --git a/README.md b/README.md index e220445c..16f61c88 100755 --- a/README.md +++ b/README.md @@ -33,7 +33,6 @@ Bench is a command-line utility that helps you to install, update, and manage mu - [Installation](#installation) - [Containerized Installation](#containerized-installation) - - [Easy Install Script](#easy-install-script) - [Manual Installation](#manual-installation) - [Usage](#basic-usage) - [Custom Bench commands](#custom-bench-commands) @@ -52,12 +51,11 @@ A typical bench setup provides two types of environments — Development and The setup for each of these installations can be achieved in multiple ways: - [Containerized Installation](#containerized-installation) - - [Easy Install Script](#easy-install-script) - [Manual Installation](#manual-installation) -We recommend using either the Docker Installation or the Easy Install Script to setup a Production Environment. For Development, you may choose either of the three methods to setup an instance. +We recommend using either the Docker Installation to setup a Production Environment. For Development, you may choose either of the two methods to setup an instance. -Otherwise, if you are looking to evaluate ERPNext, you can register for [a free trial on erpnext.com](https://erpnext.com/pricing). +Otherwise, if you are looking to evaluate Frappe apps without hassle of hosting, you can try them [on frappecloud.com](https://frappecloud.com/). ### Containerized Installation @@ -73,53 +71,6 @@ $ cd frappe_docker A quick setup guide for both the environments can be found below. For more details, check out the [Frappe/ERPNext Docker Repository](https://github.com/frappe/frappe_docker). -### Easy Install Script - -The Easy Install script should get you going with a Frappe/ERPNext setup with minimal manual intervention and effort. Since there are a lot of configurations being automatically setup, we recommend executing this script on a fresh server. - -**Note:** This script works only on GNU/Linux based server distributions, and has been designed and tested to work on Ubuntu 16.04+, CentOS 7+, and Debian-based systems. - -> This script installs Version 12 by default. It is untested with Version 13 and above. Containerized or manual installs are recommended for newer setups. - -#### Prerequisites - -You need to install the following packages for the script to run: - - - ##### Ubuntu and Debian-based Distributions: - - ```sh - $ apt install python3-minimal build-essential python3-setuptools - ``` - - - ##### CentOS and other RPM Distributions: - - ```sh - $ dnf groupinstall "Development Tools" - $ dnf install python3 - ``` - -#### Setup - -Download the Easy Install script and execute it: - -```sh -$ wget https://raw.githubusercontent.com/frappe/bench/develop/install.py -$ python3 install.py --production -``` - -The script should then prompt you for the MySQL root password and an Administrator password for the Frappe/ERPNext instance, which will then be saved under `$HOME/passwords.txt` of the user used to setup the instance. This script will then install the required stack, setup bench and a default ERPNext instance. - -When the setup is complete, you will be able to access the system at `http://`, wherein you can use the administrator password to login. - -#### Troubleshooting - -In case the setup fails, the log file is saved under `/tmp/logs/install_bench.log`. You may then: - - - Create an Issue in this repository with the log file attached. - - Search for an existing issue or post the log file on the [Frappe/ERPNext Discuss Forum](https://discuss.erpnext.com/c/bench) with the tag `installation_problem` under "Install/Update" category. - -For more information and advanced setup instructions, check out the [Easy Install Documentation](https://github.com/frappe/bench/blob/develop/docs/easy_install.md). - ### Manual Installation @@ -132,11 +83,6 @@ You'll have to set up the system dependencies required for setting up a Frappe E $ pip install frappe-bench ``` -For more extensive distribution-dependent documentation, check out the following guides: - - - [Hitchhiker's Guide to Installing Frappe on Linux](https://github.com/frappe/frappe/wiki/The-Hitchhiker%27s-Guide-to-Installing-Frappe-on-Linux) - - [Hitchhiker's Guide to Installing Frappe on MacOS](https://github.com/frappe/bench/wiki/Setting-up-a-Mac-for-Frappe-ERPNext-Development) - ## Basic Usage @@ -192,22 +138,6 @@ For more in-depth information on commands and their usage, follow [Commands and If you wish to extend the capabilities of bench with your own custom Frappe Application, you may follow [Adding Custom Bench Commands](https://github.com/frappe/bench/blob/develop/docs/bench_custom_cmd.md). -## Bench Manager - -[Bench Manager](https://github.com/frappe/bench_manager) is a GUI frontend for Bench with the same functionalties. You can install it by executing the following command: - -```sh -$ bench setup manager -``` - - - **Note:** This will create a new site to setup Bench Manager, if you want to set it up on an existing site, run the following commands: - - ```sh - $ bench get-app https://github.com/frappe/bench_manager.git - $ bench --site install-app bench_manager - ``` - - ## Guides - [Configuring HTTPS](https://frappe.io/docs/user/en/bench/guides/configuring-https.html) From 738d623117ec58d86916231e2b22855135ccf75d Mon Sep 17 00:00:00 2001 From: Ankush Menat Date: Thu, 27 Oct 2022 16:16:23 +0530 Subject: [PATCH 2/9] chore: add editorconfig --- .editorconfig | 15 +++++++++++++++ 1 file changed, 15 insertions(+) create mode 100644 .editorconfig diff --git a/.editorconfig b/.editorconfig new file mode 100644 index 00000000..a3b1ef09 --- /dev/null +++ b/.editorconfig @@ -0,0 +1,15 @@ +# Root editor config file +root = true + +# Common settings +[*] +end_of_line = lf +insert_final_newline = true +trim_trailing_whitespace = true +charset = utf-8 + +# python, js indentation settings +[{*.py,*.js,*.vue,*.css,*.scss,*.html}] +indent_style = tab +indent_size = 4 +max_line_length = 99 From 965e178e8329c3af3defed984f79354d5a7ffbd7 Mon Sep 17 00:00:00 2001 From: Ameen Ahmed Date: Thu, 17 Nov 2022 16:26:31 +0300 Subject: [PATCH 3/9] fix: FileNotFound bug (#1383) --- bench/app.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/bench/app.py b/bench/app.py index 0a13d4bd..07eb4799 100755 --- a/bench/app.py +++ b/bench/app.py @@ -198,7 +198,7 @@ class App(AppMeta): @step(title="Archiving App {repo}", success="App {repo} Archived") def remove(self, no_backup: bool = False): - active_app_path = os.path.join("apps", self.name) + active_app_path = os.path.join("apps", self.repo) if no_backup: if not os.path.islink(active_app_path): From b57838f36615b659930a4f38cfd14375097a8ad1 Mon Sep 17 00:00:00 2001 From: Ankush Menat Date: Tue, 22 Nov 2022 12:16:54 +0530 Subject: [PATCH 4/9] feat: add `max_requests` to gunicorn args As gunicorn is long running process, potentially running for days without restart the workers might start accumulating garbage that's never cleaned up and memory usage spikes after some use. This largely happens because of third-party module imports like pandas, openpyxl, numpy etc. All of these are required only for few requests and can be easily re-loaded when required. `max_requests` restarts the worker after processing number of configured requests. How to use? - If you have more than 1 gunicorn workers then this is automatically enabled. You can tweak the max_requests parameter with `gunicorn_max_requests` key in common_site_config - If you just have 1 gunicorn worker (not recommended) then this is not automatically enabled as restarting the only worker can cause spikes in response times whenever restart is triggered. --- bench/config/common_site_config.py | 14 ++++++++++++++ bench/config/supervisor.py | 11 +++++++---- bench/config/systemd.py | 11 +++++++---- bench/config/templates/supervisor.conf | 2 +- .../systemd/frappe-bench-frappe-web.service | 2 +- 5 files changed, 30 insertions(+), 10 deletions(-) diff --git a/bench/config/common_site_config.py b/bench/config/common_site_config.py index d278dedb..41242aad 100644 --- a/bench/config/common_site_config.py +++ b/bench/config/common_site_config.py @@ -15,6 +15,7 @@ default_config = { "live_reload": True, } +DEFAULT_MAX_REQUESTS = 5000 def setup_config(bench_path): make_pid_folder(bench_path) @@ -61,6 +62,19 @@ def get_gunicorn_workers(): return {"gunicorn_workers": multiprocessing.cpu_count() * 2 + 1} +def compute_max_requests_jitter(max_requests: int) -> int: + return int(max_requests * 0.1) + +def get_default_max_requests(worker_count: int): + """Get max requests and jitter config based on number of available workers.""" + + if worker_count <= 1: + # If there's only one worker then random restart can cause spikes in response times and + # can be annoying. Hence not enabled by default. + return 0 + return DEFAULT_MAX_REQUESTS + + def update_config_for_frappe(config, bench_path): ports = make_ports(bench_path) diff --git a/bench/config/supervisor.py b/bench/config/supervisor.py index d38da668..58015b81 100644 --- a/bench/config/supervisor.py +++ b/bench/config/supervisor.py @@ -8,7 +8,7 @@ import bench from bench.app import use_rq from bench.utils import get_bench_name, which from bench.bench import Bench -from bench.config.common_site_config import update_config, get_gunicorn_workers +from bench.config.common_site_config import update_config, get_gunicorn_workers, get_default_max_requests, compute_max_requests_jitter # imports - third party imports import click @@ -26,6 +26,9 @@ def generate_supervisor_config(bench_path, user=None, yes=False, skip_redis=Fals template = bench.config.env().get_template("supervisor.conf") bench_dir = os.path.abspath(bench_path) + web_worker_count = config.get("gunicorn_workers", get_gunicorn_workers()["gunicorn_workers"]) + max_requests = config.get("gunicorn_max_requests", get_default_max_requests(web_worker_count)) + config = template.render( **{ "bench_dir": bench_dir, @@ -39,9 +42,9 @@ def generate_supervisor_config(bench_path, user=None, yes=False, skip_redis=Fals "redis_socketio_config": os.path.join(bench_dir, "config", "redis_socketio.conf"), "redis_queue_config": os.path.join(bench_dir, "config", "redis_queue.conf"), "webserver_port": config.get("webserver_port", 8000), - "gunicorn_workers": config.get( - "gunicorn_workers", get_gunicorn_workers()["gunicorn_workers"] - ), + "gunicorn_workers": web_worker_count, + "gunicorn_max_requests": max_requests, + "gunicorn_max_requests_jitter": compute_max_requests_jitter(max_requests), "bench_name": get_bench_name(bench_path), "background_workers": config.get("background_workers") or 1, "bench_cmd": which("bench"), diff --git a/bench/config/systemd.py b/bench/config/systemd.py index d30edfc9..a677391c 100644 --- a/bench/config/systemd.py +++ b/bench/config/systemd.py @@ -9,7 +9,7 @@ import click import bench from bench.app import use_rq from bench.bench import Bench -from bench.config.common_site_config import get_gunicorn_workers, update_config +from bench.config.common_site_config import get_gunicorn_workers, update_config, get_default_max_requests, compute_max_requests_jitter from bench.utils import exec_cmd, which, get_bench_name @@ -61,6 +61,9 @@ def generate_systemd_config( get_bench_name(bench_path) + "-frappe-long-worker@" + str(i + 1) + ".service" ) + web_worker_count = config.get("gunicorn_workers", get_gunicorn_workers()["gunicorn_workers"]) + max_requests = config.get("gunicorn_max_requests", get_default_max_requests(web_worker_count)) + bench_info = { "bench_dir": bench_dir, "sites_dir": os.path.join(bench_dir, "sites"), @@ -73,9 +76,9 @@ def generate_systemd_config( "redis_socketio_config": os.path.join(bench_dir, "config", "redis_socketio.conf"), "redis_queue_config": os.path.join(bench_dir, "config", "redis_queue.conf"), "webserver_port": config.get("webserver_port", 8000), - "gunicorn_workers": config.get( - "gunicorn_workers", get_gunicorn_workers()["gunicorn_workers"] - ), + "gunicorn_workers": web_worker_count, + "gunicorn_max_requests": max_requests, + "gunicorn_max_requests_jitter": compute_max_requests_jitter(max_requests), "bench_name": get_bench_name(bench_path), "worker_target_wants": " ".join(background_workers), "bench_cmd": which("bench"), diff --git a/bench/config/templates/supervisor.conf b/bench/config/templates/supervisor.conf index 085cc2cf..f29c1673 100644 --- a/bench/config/templates/supervisor.conf +++ b/bench/config/templates/supervisor.conf @@ -3,7 +3,7 @@ ; killasgroup=true --> send kill signal to child processes too [program:{{ bench_name }}-frappe-web] -command={{ bench_dir }}/env/bin/gunicorn -b 127.0.0.1:{{ webserver_port }} -w {{ gunicorn_workers }} -t {{ http_timeout }} frappe.app:application --preload +command={{ bench_dir }}/env/bin/gunicorn -b 127.0.0.1:{{ webserver_port }} -w {{ gunicorn_workers }} --max-requests {{ gunicorn_max_requests }} --max-requests-jitter {{ gunicorn_max_requests_jitter }} -t {{ http_timeout }} frappe.app:application --preload priority=4 autostart=true autorestart=true diff --git a/bench/config/templates/systemd/frappe-bench-frappe-web.service b/bench/config/templates/systemd/frappe-bench-frappe-web.service index bb2f0e38..0621e1d3 100644 --- a/bench/config/templates/systemd/frappe-bench-frappe-web.service +++ b/bench/config/templates/systemd/frappe-bench-frappe-web.service @@ -6,7 +6,7 @@ PartOf={{ bench_name }}-web.target User={{ user }} Group={{ user }} Restart=always -ExecStart={{ bench_dir }}/env/bin/gunicorn -b 127.0.0.1:{{ webserver_port }} -w {{ gunicorn_workers }} -t {{ http_timeout }} frappe.app:application --preload +ExecStart={{ bench_dir }}/env/bin/gunicorn -b 127.0.0.1:{{ webserver_port }} -w {{ gunicorn_workers }} -t {{ http_timeout }} --max-requests {{ gunicorn_max_requests }} --max-requests-jitter {{ gunicorn_max_requests_jitter }} frappe.app:application --preload StandardOutput=file:{{ bench_dir }}/logs/web.log StandardError=file:{{ bench_dir }}/logs/web.error.log WorkingDirectory={{ sites_dir }} From d3cb7eceb40bbfda86382fee70e0b58010dd7d05 Mon Sep 17 00:00:00 2001 From: Ankush Menat Date: Wed, 23 Nov 2022 11:47:09 +0530 Subject: [PATCH 5/9] ci: fix flake8 url --- .pre-commit-config.yaml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index b40aa088..d608bf4a 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -29,8 +29,8 @@ repos: - id: black additional_dependencies: ['click==8.0.4'] - - repo: https://gitlab.com/pycqa/flake8 - rev: 3.9.2 + - repo: https://github.com/pycqa/flake8 + rev: 5.0.4 hooks: - id: flake8 additional_dependencies: ['flake8-bugbear',] From f45db01d9aaff6ab19e3cf9eeed6c2708313c957 Mon Sep 17 00:00:00 2001 From: Ankush Menat Date: Wed, 23 Nov 2022 11:54:58 +0530 Subject: [PATCH 6/9] chore: remove deprecated easy install script --- docs/easy_install.md | 93 -------- install.py | 499 ------------------------------------------- 2 files changed, 592 deletions(-) delete mode 100644 docs/easy_install.md delete mode 100644 install.py diff --git a/docs/easy_install.md b/docs/easy_install.md deleted file mode 100644 index 91ee5566..00000000 --- a/docs/easy_install.md +++ /dev/null @@ -1,93 +0,0 @@ -# Easy Install Script - -- This script will install the pre-requisites, install bench and setup an ERPNext site `(site1.local under frappe-bench)` -- Passwords for Frappe Administrator and MariaDB (root) will be asked and saved under `~/passwords.txt` -- MariaDB (root) password may be `password` on a fresh server -- You can then login as **Administrator** with the Administrator password -- The log file is saved under `/tmp/logs/install_bench.log` in case you run into any issues during the install. -- If you find any problems, post them on the forum: [https://discuss.erpnext.com](https://discuss.erpnext.com/tags/installation_problem) under the "Install / Update" category. - ---- - -## What will this script do? - -- Install all the pre-requisites -- Install the command line `bench` (under ~/.bench) -- Create a new bench (a folder that will contain your entire frappe/erpnext setup at ~/frappe-bench) -- Create a new ERPNext site on the bench (site1.local) - ---- - -## Getting started with easy install... - -Open your Terminal and enter: - -#### 0. Setup user & Download the install script - -If you are on a fresh server and logged in as root, at first create a dedicated user for frappe -& equip this user with sudo privileges - -``` - adduser [frappe-user] - usermod -aG sudo [frappe-user] -``` - -*(it is very common to use "frappe" as frappe-username, but this comes with the security flaw of ["frappe" ranking very high](https://www.reddit.com/r/dataisbeautiful/comments/b3sirt/i_deployed_over_a_dozen_cyber_honeypots_all_over/?st=JTJ0SC0Q&sh=76e05240) in as a username challenged in hacking attempts. So, for production sites it is highly recommended to use a custom username harder to guess)* - -*(you can specify the flag --home to specify a directory for your [frappe-user]. Bench will follow the home directory specified by the user's home directory e.g. /data/[frappe-user]/frappe-bench)* - -Switch to `[frappe-user]` (using `su [frappe-user]`) and start the setup - - wget https://raw.githubusercontent.com/frappe/bench/develop/install.py - - -#### 1. Run the install script - - sudo python3 install.py - -*Note: `user` flag to create a user and install using that user (By default, the script will create a user with the username `frappe` if the --user flag is not used)* - -For production or development, append the `--production` or `--develop` flag to the command respectively. - - sudo python3 install.py --production --user [frappe-user] - -or - - sudo python3 install.py --develop - sudo python3 install.py --develop --user [frappe-user] - - sudo python3 install.py --production --user [frappe-user] --container - -*Note: `container` flag to install inside a container (this will prevent the `/proc/sys/vm/swappiness: Read-only` file system error)* - - - python3 install.py --production --version 11 --user [frappe-user] - -use --version flag to install specific version - - python3 install.py --production --version 11 --python python2.7 --user [frappe-user] - -use --python flag to specify virtual environments python version, by default script setup python3 - ---- - -## How do I start ERPNext - -1. For development: Go to your bench folder (`~[frappe-user]/frappe-bench` by default) and start the bench with `bench start` -2. For production: Your process will be setup and managed by `nginx` and `supervisor`. Checkout [Setup Production](https://frappe.io/docs/user/en/bench/guides/setup-production.html) for more information. - ---- - -## An error occured mid installation? - -TLDR; Save the logs! - -1. The easy install script starts multiple processes to install prerequisites, system dependencies, requirements, sets up locales, configuration files, etc. - -2. The script pipes all these process outputs and saves it under `/tmp/log/{easy-install-filename}.log` as prompted by the script in the beginning of the script or/and if something went wrong again. - -3. Retain this log file and share it in case you need help with proceeding with the install. Since, the file's saved under `/tmp` it'll be cleared by the system after a reboot. Be careful to save it elsewhere if needed! - -3. A lot of things can go wrong in setting up the environment due to prior settings, company protocols or even breaking changes in system packages and their dependencies. - -4. Sharing your logfile in any issues opened related to this can help us find solutions to it faster and make the script better! diff --git a/install.py b/install.py deleted file mode 100644 index 54248b9c..00000000 --- a/install.py +++ /dev/null @@ -1,499 +0,0 @@ -#!/usr/bin/env python3 - -from __future__ import print_function - -import os -import sys -import subprocess -import getpass -import json -import multiprocessing -import shutil -import platform -import warnings -import datetime - - -tmp_bench_repo = os.path.join('/', 'tmp', '.bench') -tmp_log_folder = os.path.join('/', 'tmp', 'logs') -execution_timestamp = datetime.datetime.utcnow() -execution_day = "{:%Y-%m-%d}".format(execution_timestamp) -execution_time = "{:%H:%M}".format(execution_timestamp) -log_file_name = "easy-install__{0}__{1}.log".format(execution_day, execution_time.replace(':', '-')) -log_path = os.path.join(tmp_log_folder, log_file_name) -log_stream = sys.stdout -distro_required = not ((sys.version_info.major < 3) or (sys.version_info.major == 3 and sys.version_info.minor < 5)) - - -def log(message, level=0): - levels = { - 0: '\033[94m', # normal - 1: '\033[92m', # success - 2: '\033[91m', # fail - 3: '\033[93m' # warn/suggest - } - start = levels.get(level) or '' - end = '\033[0m' - print(start + message + end) - - -def setup_log_stream(args): - global log_stream - sys.stderr = sys.stdout - - if not args.verbose: - if not os.path.exists(tmp_log_folder): - os.makedirs(tmp_log_folder) - log_stream = open(log_path, 'w') - log("Logs are saved under {0}".format(log_path), level=3) - print("Install script run at {0} on {1}\n\n".format(execution_time, execution_day), file=log_stream) - - -def check_environment(): - needed_environ_vars = ['LANG', 'LC_ALL'] - message = '' - - for var in needed_environ_vars: - if var not in os.environ: - message += "\nexport {0}=C.UTF-8".format(var) - - if message: - log("Bench's CLI needs these to be defined!", level=3) - log("Run the following commands in shell: {0}".format(message), level=2) - sys.exit() - - -def check_system_package_managers(): - if 'Darwin' in os.uname(): - if not shutil.which('brew'): - raise Exception(''' - Please install brew package manager before proceeding with bench setup. Please run following - to install brew package manager on your machine, - - /usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)" - ''') - if 'Linux' in os.uname(): - if not any([shutil.which(x) for x in ['apt-get', 'yum']]): - raise Exception('Cannot find any compatible package manager!') - - -def check_distribution_compatibility(): - dist_name, dist_version = get_distribution_info() - supported_dists = { - 'macos': [10.9, 10.10, 10.11, 10.12], - 'ubuntu': [14, 15, 16, 18, 19, 20], - 'debian': [8, 9, 10], - 'centos': [7] - } - - log("Checking System Compatibility...") - if dist_name in supported_dists: - if float(dist_version) in supported_dists[dist_name]: - log("{0} {1} is compatible!".format(dist_name, dist_version), level=1) - else: - log("{0} {1} is detected".format(dist_name, dist_version), level=1) - log("Install on {0} {1} instead".format(dist_name, supported_dists[dist_name][-1]), level=3) - else: - log("Sorry, the installer doesn't support {0}. Aborting installation!".format(dist_name), level=2) - - -def import_with_install(package): - # copied from https://discuss.erpnext.com/u/nikunj_patel - # https://discuss.erpnext.com/t/easy-install-setup-guide-for-erpnext-installation-on-ubuntu-20-04-lts-with-some-modification-of-course/62375/5 - # need to move to top said v13 for fully python3 era - import importlib - - try: - importlib.import_module(package) - except ImportError: - # caveat : pip3 must be installed - - import pip - - pip.main(['install', package]) - finally: - globals()[package] = importlib.import_module(package) - - -def get_distribution_info(): - # return distribution name and major version - if platform.system() == "Linux": - if distro_required: - current_dist = distro.linux_distribution(full_distribution_name=True) - else: - current_dist = platform.dist() - - return current_dist[0].lower(), current_dist[1].rsplit('.')[0] - - elif platform.system() == "Darwin": - current_dist = platform.mac_ver() - return "macos", current_dist[0].rsplit('.', 1)[0] - - -def run_os_command(command_map): - '''command_map is a dictionary of {'executable': command}. For ex. {'apt-get': 'sudo apt-get install -y python2.7'}''' - success = True - - for executable, commands in command_map.items(): - if shutil.which(executable): - if isinstance(commands, str): - commands = [commands] - - for command in commands: - returncode = subprocess.check_call(command, shell=True, stdout=log_stream, stderr=sys.stderr) - success = success and (returncode == 0) - - return success - - -def install_prerequisites(): - # pre-requisites for bench repo cloning - run_os_command({ - 'apt-get': [ - 'sudo apt-get update', - 'sudo apt-get install -y git build-essential python3-setuptools python3-dev libffi-dev' - ], - 'yum': [ - 'sudo yum groupinstall -y "Development tools"', - 'sudo yum install -y epel-release redhat-lsb-core git python-setuptools python-devel openssl-devel libffi-devel' - ] - }) - - # until psycopg2-binary is available for aarch64 (Arm 64-bit), we'll need libpq and libssl dev packages to build psycopg2 from source - if platform.machine() == 'aarch64': - log("Installing libpq and libssl dev packages to build psycopg2 for aarch64...") - run_os_command({ - 'apt-get': ['sudo apt-get install -y libpq-dev libssl-dev'], - 'yum': ['sudo yum install -y libpq-devel openssl-devel'] - }) - - install_package('curl') - install_package('wget') - install_package('git') - install_package('pip3', 'python3-pip') - - run_os_command({ - 'python3': "sudo -H python3 -m pip install --upgrade pip setuptools-rust" - }) - success = run_os_command({ - 'python3': "sudo -H python3 -m pip install --upgrade setuptools wheel cryptography ansible~=2.8.15" - }) - - if not (success or shutil.which('ansible')): - could_not_install('Ansible') - - -def could_not_install(package): - raise Exception('Could not install {0}. Please install it manually.'.format(package)) - - -def is_sudo_user(): - return os.geteuid() == 0 - - -def install_package(package, package_name=None): - if shutil.which(package): - log("{0} already installed!".format(package), level=1) - else: - log("Installing {0}...".format(package)) - package_name = package_name or package - success = run_os_command({ - 'apt-get': ['sudo apt-get install -y {0}'.format(package_name)], - 'yum': ['sudo yum install -y {0}'.format(package_name)], - 'brew': ['brew install {0}'.format(package_name)] - }) - if success: - log("{0} installed!".format(package), level=1) - return success - could_not_install(package) - - -def install_bench(args): - # clone bench repo - if not args.run_travis: - clone_bench_repo(args) - - if not args.user: - if args.production: - args.user = 'frappe' - - elif 'SUDO_USER' in os.environ: - args.user = os.environ['SUDO_USER'] - - else: - args.user = getpass.getuser() - - if args.user == 'root': - raise Exception('Please run this script as a non-root user with sudo privileges, but without using sudo or pass --user=USER') - - # Python executable - dist_name, dist_version = get_distribution_info() - if dist_name=='centos': - args.python = 'python3.6' - else: - args.python = 'python3' - - # create user if not exists - extra_vars = vars(args) - extra_vars.update(frappe_user=args.user) - - extra_vars.update(user_directory=get_user_home_directory(args.user)) - - if os.path.exists(tmp_bench_repo): - repo_path = tmp_bench_repo - else: - repo_path = os.path.join(os.path.expanduser('~'), 'bench') - - extra_vars.update(repo_path=repo_path) - run_playbook('create_user.yml', extra_vars=extra_vars) - - extra_vars.update(get_passwords(args)) - if args.production: - extra_vars.update(max_worker_connections=multiprocessing.cpu_count() * 1024) - - if args.version <= 10: - frappe_branch = "{0}.x.x".format(args.version) - erpnext_branch = "{0}.x.x".format(args.version) - else: - frappe_branch = "version-{0}".format(args.version) - erpnext_branch = "version-{0}".format(args.version) - - # Allow override of frappe_branch and erpnext_branch, regardless of args.version (which always has a default set) - if args.frappe_branch: - frappe_branch = args.frappe_branch - if args.erpnext_branch: - erpnext_branch = args.erpnext_branch - - extra_vars.update(frappe_branch=frappe_branch) - extra_vars.update(erpnext_branch=erpnext_branch) - - bench_name = 'frappe-bench' if not args.bench_name else args.bench_name - extra_vars.update(bench_name=bench_name) - - # Will install ERPNext production setup by default - if args.without_erpnext: - log("Initializing bench {bench_name}:\n\tFrappe Branch: {frappe_branch}\n\tERPNext will not be installed due to --without-erpnext".format(bench_name=bench_name, frappe_branch=frappe_branch)) - else: - log("Initializing bench {bench_name}:\n\tFrappe Branch: {frappe_branch}\n\tERPNext Branch: {erpnext_branch}".format(bench_name=bench_name, frappe_branch=frappe_branch, erpnext_branch=erpnext_branch)) - run_playbook('site.yml', sudo=True, extra_vars=extra_vars) - - if os.path.exists(tmp_bench_repo): - shutil.rmtree(tmp_bench_repo) - - -def clone_bench_repo(args): - '''Clones the bench repository in the user folder''' - branch = args.bench_branch or 'develop' - repo_url = args.repo_url or 'https://github.com/frappe/bench' - - if os.path.exists(tmp_bench_repo): - log('Not cloning already existing Bench repository at {tmp_bench_repo}'.format(tmp_bench_repo=tmp_bench_repo)) - return 0 - elif args.without_bench_setup: - clone_path = os.path.join(os.path.expanduser('~'), 'bench') - log('--without-bench-setup specified, clone path is: {clone_path}'.format(clone_path=clone_path)) - else: - clone_path = tmp_bench_repo - # Not logging repo_url to avoid accidental credential leak in case credential is embedded in URL - log('Cloning bench repository branch {branch} into {clone_path}'.format(branch=branch, clone_path=clone_path)) - - success = run_os_command( - {'git': 'git clone --quiet {repo_url} {bench_repo} --depth 1 --branch {branch}'.format( - repo_url=repo_url, bench_repo=clone_path, branch=branch)} - ) - - return success - - -def passwords_didnt_match(context=''): - log("{} passwords did not match!".format(context), level=3) - - -def get_passwords(args): - """ - Returns a dict of passwords for further use - and creates passwords.txt in the bench user's home directory - """ - log("Input MySQL and Frappe Administrator passwords:") - ignore_prompt = args.run_travis or args.without_bench_setup - mysql_root_password, admin_password = '', '' - passwords_file_path = os.path.join(os.path.expanduser('~' + args.user), 'passwords.txt') - - if not ignore_prompt: - # set passwords from existing passwords.txt - if os.path.isfile(passwords_file_path): - with open(passwords_file_path, 'r') as f: - passwords = json.load(f) - mysql_root_password, admin_password = passwords['mysql_root_password'], passwords['admin_password'] - - # set passwords from cli args - if args.mysql_root_password: - mysql_root_password = args.mysql_root_password - if args.admin_password: - admin_password = args.admin_password - - # prompt for passwords - pass_set = True - while pass_set: - # mysql root password - if not mysql_root_password: - mysql_root_password = getpass.unix_getpass(prompt='Please enter mysql root password: ') - conf_mysql_passwd = getpass.unix_getpass(prompt='Re-enter mysql root password: ') - - if mysql_root_password != conf_mysql_passwd or mysql_root_password == '': - passwords_didnt_match("MySQL") - mysql_root_password = '' - continue - - # admin password, only needed if we're also creating a site - if not admin_password and not args.without_site: - admin_password = getpass.unix_getpass(prompt='Please enter the default Administrator user password: ') - conf_admin_passswd = getpass.unix_getpass(prompt='Re-enter Administrator password: ') - - if admin_password != conf_admin_passswd or admin_password == '': - passwords_didnt_match("Administrator") - admin_password = '' - continue - elif args.without_site: - log("Not creating a new site due to --without-site") - - pass_set = False - else: - mysql_root_password = admin_password = 'travis' - - passwords = { - 'mysql_root_password': mysql_root_password, - 'admin_password': admin_password - } - - if not ignore_prompt: - with open(passwords_file_path, 'w') as f: - json.dump(passwords, f, indent=1) - - log('Passwords saved at ~/passwords.txt') - - return passwords - - -def get_extra_vars_json(extra_args): - # We need to pass production as extra_vars to the playbook to execute conditionals in the - # playbook. Extra variables can passed as json or key=value pair. Here, we will use JSON. - json_path = os.path.join('/', 'tmp', 'extra_vars.json') - extra_vars = dict(list(extra_args.items())) - - with open(json_path, mode='w') as j: - json.dump(extra_vars, j, indent=1, sort_keys=True) - - return ('@' + json_path) - -def get_user_home_directory(user): - # Return home directory /home/USERNAME or anything else defined as home directory in - # passwd for user. - return os.path.expanduser('~'+user) - - -def run_playbook(playbook_name, sudo=False, extra_vars=None): - args = ['ansible-playbook', '-c', 'local', playbook_name , '-vvvv'] - - if extra_vars: - args.extend(['-e', get_extra_vars_json(extra_vars)]) - - if sudo: - user = extra_vars.get('user') or getpass.getuser() - args.extend(['--become', '--become-user={0}'.format(user)]) - - if os.path.exists(tmp_bench_repo): - cwd = tmp_bench_repo - else: - cwd = os.path.join(os.path.expanduser('~'), 'bench') - - playbooks_locations = [os.path.join(cwd, 'bench', 'playbooks'), os.path.join(cwd, 'playbooks')] - playbooks_folder = [x for x in playbooks_locations if os.path.exists(x)][0] - - success = subprocess.check_call(args, cwd=playbooks_folder, stdout=log_stream, stderr=sys.stderr) - return success - - -def setup_script_requirements(): - if distro_required: - install_package('pip3', 'python3-pip') - import_with_install('distro') - - -def parse_commandline_args(): - import argparse - - parser = argparse.ArgumentParser(description='Frappe Installer') - # Arguments develop and production are mutually exclusive both can't be specified together. - # Hence, we need to create a group for discouraging use of both options at the same time. - args_group = parser.add_mutually_exclusive_group() - - args_group.add_argument('--develop', dest='develop', action='store_true', default=False, help='Install developer setup') - args_group.add_argument('--production', dest='production', action='store_true', default=False, help='Setup Production environment for bench') - parser.add_argument('--site', dest='site', action='store', default='site1.local', help='Specify name for your first ERPNext site') - parser.add_argument('--without-site', dest='without_site', action='store_true', default=False, help='Do not create a new site') - parser.add_argument('--verbose', dest='verbose', action='store_true', default=False, help='Run the script in verbose mode') - parser.add_argument('--user', dest='user', help='Install frappe-bench for this user') - parser.add_argument('--bench-branch', dest='bench_branch', help='Clone a particular branch of bench repository') - parser.add_argument('--repo-url', dest='repo_url', help='Clone bench from the given url') - parser.add_argument('--frappe-repo-url', dest='frappe_repo_url', action='store', default='https://github.com/frappe/frappe', help='Clone frappe from the given url') - parser.add_argument('--frappe-branch', dest='frappe_branch', action='store', help='Clone a particular branch of frappe') - parser.add_argument('--erpnext-repo-url', dest='erpnext_repo_url', action='store', default='https://github.com/frappe/erpnext', help='Clone erpnext from the given url') - parser.add_argument('--erpnext-branch', dest='erpnext_branch', action='store', help='Clone a particular branch of erpnext') - parser.add_argument('--without-erpnext', dest='without_erpnext', action='store_true', default=False, help='Prevent fetching ERPNext') - # direct provision to install versions - parser.add_argument('--version', dest='version', action='store', default=13, type=int, help='Clone particular version of frappe and erpnext') - # To enable testing of script using Travis, this should skip the prompt - parser.add_argument('--run-travis', dest='run_travis', action='store_true', default=False, help=argparse.SUPPRESS) - parser.add_argument('--without-bench-setup', dest='without_bench_setup', action='store_true', default=False, help=argparse.SUPPRESS) - # whether to overwrite an existing bench - parser.add_argument('--overwrite', dest='overwrite', action='store_true', default=False, help='Whether to overwrite an existing bench') - # set passwords - parser.add_argument('--mysql-root-password', dest='mysql_root_password', help='Set mysql root password') - parser.add_argument('--mariadb-version', dest='mariadb_version', default='10.4', help='Specify mariadb version') - parser.add_argument('--admin-password', dest='admin_password', help='Set admin password') - parser.add_argument('--bench-name', dest='bench_name', help='Create bench with specified name. Default name is frappe-bench') - # Python interpreter to be used - parser.add_argument('--python', dest='python', default='python3', help=argparse.SUPPRESS) - # LXC Support - parser.add_argument('--container', dest='container', default=False, action='store_true', help='Use if you\'re creating inside LXC') - - args = parser.parse_args() - - return args - - -if __name__ == '__main__': - if sys.version[0] == '2': - if not os.environ.get('CI'): - if not raw_input("It is recommended to run this script with Python 3\nDo you still wish to continue? [Y/n]: ").lower() == "y": - sys.exit() - - try: - from distutils.spawn import find_executable - except ImportError: - try: - subprocess.check_call('pip install --upgrade setuptools') - except subprocess.CalledProcessError: - print("Install distutils or use Python3 to run the script") - sys.exit(1) - - shutil.which = find_executable - - if not is_sudo_user(): - log("Please run this script as a non-root user with sudo privileges", level=3) - sys.exit() - - args = parse_commandline_args() - - with warnings.catch_warnings(): - warnings.simplefilter("ignore") - setup_log_stream(args) - install_prerequisites() - setup_script_requirements() - check_distribution_compatibility() - check_system_package_managers() - check_environment() - install_bench(args) - - log("Bench + Frappe + ERPNext has been successfully installed!") From c4305fd528f30765e2b8a59c4979ff286433482c Mon Sep 17 00:00:00 2001 From: Ankush Menat Date: Wed, 23 Nov 2022 14:35:01 +0530 Subject: [PATCH 7/9] style: format everything w black --- bench/config/common_site_config.py | 4 +++- bench/config/supervisor.py | 15 ++++++++++++--- bench/config/systemd.py | 15 ++++++++++++--- bench/utils/__init__.py | 5 ++++- bench/utils/app.py | 4 +++- 5 files changed, 34 insertions(+), 9 deletions(-) diff --git a/bench/config/common_site_config.py b/bench/config/common_site_config.py index 41242aad..df58daf1 100644 --- a/bench/config/common_site_config.py +++ b/bench/config/common_site_config.py @@ -17,6 +17,7 @@ default_config = { DEFAULT_MAX_REQUESTS = 5000 + def setup_config(bench_path): make_pid_folder(bench_path) bench_config = get_config(bench_path) @@ -62,9 +63,11 @@ def get_gunicorn_workers(): return {"gunicorn_workers": multiprocessing.cpu_count() * 2 + 1} + def compute_max_requests_jitter(max_requests: int) -> int: return int(max_requests * 0.1) + def get_default_max_requests(worker_count: int): """Get max requests and jitter config based on number of available workers.""" @@ -75,7 +78,6 @@ def get_default_max_requests(worker_count: int): return DEFAULT_MAX_REQUESTS - def update_config_for_frappe(config, bench_path): ports = make_ports(bench_path) diff --git a/bench/config/supervisor.py b/bench/config/supervisor.py index 58015b81..b78c0470 100644 --- a/bench/config/supervisor.py +++ b/bench/config/supervisor.py @@ -8,7 +8,12 @@ import bench from bench.app import use_rq from bench.utils import get_bench_name, which from bench.bench import Bench -from bench.config.common_site_config import update_config, get_gunicorn_workers, get_default_max_requests, compute_max_requests_jitter +from bench.config.common_site_config import ( + update_config, + get_gunicorn_workers, + get_default_max_requests, + compute_max_requests_jitter, +) # imports - third party imports import click @@ -26,8 +31,12 @@ def generate_supervisor_config(bench_path, user=None, yes=False, skip_redis=Fals template = bench.config.env().get_template("supervisor.conf") bench_dir = os.path.abspath(bench_path) - web_worker_count = config.get("gunicorn_workers", get_gunicorn_workers()["gunicorn_workers"]) - max_requests = config.get("gunicorn_max_requests", get_default_max_requests(web_worker_count)) + web_worker_count = config.get( + "gunicorn_workers", get_gunicorn_workers()["gunicorn_workers"] + ) + max_requests = config.get( + "gunicorn_max_requests", get_default_max_requests(web_worker_count) + ) config = template.render( **{ diff --git a/bench/config/systemd.py b/bench/config/systemd.py index a677391c..0d4e1243 100644 --- a/bench/config/systemd.py +++ b/bench/config/systemd.py @@ -9,7 +9,12 @@ import click import bench from bench.app import use_rq from bench.bench import Bench -from bench.config.common_site_config import get_gunicorn_workers, update_config, get_default_max_requests, compute_max_requests_jitter +from bench.config.common_site_config import ( + get_gunicorn_workers, + update_config, + get_default_max_requests, + compute_max_requests_jitter, +) from bench.utils import exec_cmd, which, get_bench_name @@ -61,8 +66,12 @@ def generate_systemd_config( get_bench_name(bench_path) + "-frappe-long-worker@" + str(i + 1) + ".service" ) - web_worker_count = config.get("gunicorn_workers", get_gunicorn_workers()["gunicorn_workers"]) - max_requests = config.get("gunicorn_max_requests", get_default_max_requests(web_worker_count)) + web_worker_count = config.get( + "gunicorn_workers", get_gunicorn_workers()["gunicorn_workers"] + ) + max_requests = config.get( + "gunicorn_max_requests", get_default_max_requests(web_worker_count) + ) bench_info = { "bench_dir": bench_dir, diff --git a/bench/utils/__init__.py b/bench/utils/__init__.py index ff8f8fcd..b61f686e 100644 --- a/bench/utils/__init__.py +++ b/bench/utils/__init__.py @@ -125,7 +125,10 @@ def check_latest_version(): local_version = Version(VERSION) if pypi_version > local_version: - log(f"A newer version of bench is available: {local_version} → {pypi_version}", stderr=True) + log( + f"A newer version of bench is available: {local_version} → {pypi_version}", + stderr=True, + ) def pause_exec(seconds=10): diff --git a/bench/utils/app.py b/bench/utils/app.py index 790792da..5541b548 100644 --- a/bench/utils/app.py +++ b/bench/utils/app.py @@ -185,7 +185,9 @@ def get_required_deps(org, name, branch, deps="hooks.py"): res = requests.get(url=git_api_url, params=params).json() if "message" in res: - git_url = f"https://raw.githubusercontent.com/{org}/{name}/{params['ref']}/{name}/{deps}" + git_url = ( + f"https://raw.githubusercontent.com/{org}/{name}/{params['ref']}/{name}/{deps}" + ) return requests.get(git_url).text return base64.decodebytes(res["content"].encode()).decode() From 9c80f5d24f3e9a88531c7300f95ee7dc3839256d Mon Sep 17 00:00:00 2001 From: Ankush Menat Date: Thu, 24 Nov 2022 15:29:34 +0530 Subject: [PATCH 8/9] perf: single worker in development install (#1392) Most developers don't need 3 separate workers in development. This changes procfile to use single worker to consume from all queues in development. Pros: - Lighter development setups Cons: - Not "equivalent to production" - not required in most cases so eh. You can still edit procfile to start whatever process you want anyway. --- bench/config/templates/Procfile | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/bench/config/templates/Procfile b/bench/config/templates/Procfile index d9391087..899d1290 100644 --- a/bench/config/templates/Procfile +++ b/bench/config/templates/Procfile @@ -11,9 +11,7 @@ watch: bench watch {% endif %} {% if use_rq -%} schedule: bench schedule -worker_short: bench worker --queue short 1>> logs/worker.log 2>> logs/worker.error.log -worker_long: bench worker --queue long 1>> logs/worker.log 2>> logs/worker.error.log -worker_default: bench worker --queue default 1>> logs/worker.log 2>> logs/worker.error.log +worker: bench worker 1>> logs/worker.log 2>> logs/worker.error.log {% for worker_name, worker_details in workers.items() %} worker_{{ worker_name }}: bench worker --queue {{ worker_name }} 1>> logs/worker.log 2>> logs/worker.error.log {% endfor %} From c59d1edee5e2c22f55f91ac946df8e09535c8927 Mon Sep 17 00:00:00 2001 From: Ankush Menat Date: Mon, 28 Nov 2022 13:05:00 +0530 Subject: [PATCH 9/9] fix: restart proc manager if set in config (#1391) --- bench/bench.py | 2 +- bench/config/systemd.py | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/bench/bench.py b/bench/bench.py index 8586c712..e799838a 100644 --- a/bench/bench.py +++ b/bench/bench.py @@ -147,7 +147,7 @@ class Bench(Base, Validator): if conf.get("developer_mode"): restart_process_manager(bench_path=self.name, web_workers=web) - if supervisor and conf.get("restart_supervisor_on_update"): + if supervisor or conf.get("restart_supervisor_on_update"): restart_supervisor_processes(bench_path=self.name, web_workers=web) if systemd and conf.get("restart_systemd_on_update"): restart_systemd_processes(bench_path=self.name, web_workers=web) diff --git a/bench/config/systemd.py b/bench/config/systemd.py index 0d4e1243..a19de662 100644 --- a/bench/config/systemd.py +++ b/bench/config/systemd.py @@ -105,7 +105,7 @@ def generate_systemd_config( setup_web_config(bench_info, bench_path) setup_redis_config(bench_info, bench_path) - update_config({"restart_systemd_on_update": True}, bench_path=bench_path) + update_config({"restart_systemd_on_update": False}, bench_path=bench_path) update_config({"restart_supervisor_on_update": False}, bench_path=bench_path)