diff --git a/.editorconfig b/.editorconfig new file mode 100644 index 00000000..a3b1ef09 --- /dev/null +++ b/.editorconfig @@ -0,0 +1,15 @@ +# Root editor config file +root = true + +# Common settings +[*] +end_of_line = lf +insert_final_newline = true +trim_trailing_whitespace = true +charset = utf-8 + +# python, js indentation settings +[{*.py,*.js,*.vue,*.css,*.scss,*.html}] +indent_style = tab +indent_size = 4 +max_line_length = 99 diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index b40aa088..d608bf4a 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -29,8 +29,8 @@ repos: - id: black additional_dependencies: ['click==8.0.4'] - - repo: https://gitlab.com/pycqa/flake8 - rev: 3.9.2 + - repo: https://github.com/pycqa/flake8 + rev: 5.0.4 hooks: - id: flake8 additional_dependencies: ['flake8-bugbear',] diff --git a/README.md b/README.md index eb1903e8..ca9d4881 100755 --- a/README.md +++ b/README.md @@ -30,7 +30,6 @@ Bench is a command-line utility that helps you to install, update, and manage mu - [Installation](#installation) - [Containerized Installation](#containerized-installation) - - [Easy Install Script](#easy-install-script) - [Manual Installation](#manual-installation) - [Usage](#basic-usage) - [Custom Bench commands](#custom-bench-commands) @@ -49,12 +48,11 @@ A typical bench setup provides two types of environments — Development and The setup for each of these installations can be achieved in multiple ways: - [Containerized Installation](#containerized-installation) - - [Easy Install Script](#easy-install-script) - [Manual Installation](#manual-installation) -We recommend using either the Docker Installation or the Easy Install Script to setup a Production Environment. For Development, you may choose either of the three methods to setup an instance. +We recommend using either the Docker Installation to setup a Production Environment. For Development, you may choose either of the two methods to setup an instance. -Otherwise, if you are looking to evaluate ERPNext, you can register for [a free trial on erpnext.com](https://erpnext.com/pricing). +Otherwise, if you are looking to evaluate Frappe apps without hassle of hosting, you can try them [on frappecloud.com](https://frappecloud.com/). ### Containerized Installation @@ -70,53 +68,6 @@ $ cd frappe_docker A quick setup guide for both the environments can be found below. For more details, check out the [Frappe/ERPNext Docker Repository](https://github.com/frappe/frappe_docker). -### Easy Install Script - -The Easy Install script should get you going with a Frappe/ERPNext setup with minimal manual intervention and effort. Since there are a lot of configurations being automatically setup, we recommend executing this script on a fresh server. - -**Note:** This script works only on GNU/Linux based server distributions, and has been designed and tested to work on Ubuntu 16.04+, CentOS 7+, and Debian-based systems. - -> This script installs Version 12 by default. It is untested with Version 13 and above. Containerized or manual installs are recommended for newer setups. - -#### Prerequisites - -You need to install the following packages for the script to run: - - - ##### Ubuntu and Debian-based Distributions: - - ```sh - $ apt install python3-minimal build-essential python3-setuptools - ``` - - - ##### CentOS and other RPM Distributions: - - ```sh - $ dnf groupinstall "Development Tools" - $ dnf install python3 - ``` - -#### Setup - -Download the Easy Install script and execute it: - -```sh -$ wget https://raw.githubusercontent.com/frappe/bench/develop/install.py -$ python3 install.py --production -``` - -The script should then prompt you for the MySQL root password and an Administrator password for the Frappe/ERPNext instance, which will then be saved under `$HOME/passwords.txt` of the user used to setup the instance. This script will then install the required stack, setup bench and a default ERPNext instance. - -When the setup is complete, you will be able to access the system at `http://`, wherein you can use the administrator password to login. - -#### Troubleshooting - -In case the setup fails, the log file is saved under `/tmp/logs/install_bench.log`. You may then: - - - Create an Issue in this repository with the log file attached. - - Search for an existing issue or post the log file on the [Frappe/ERPNext Discuss Forum](https://discuss.erpnext.com/c/bench) with the tag `installation_problem` under "Install/Update" category. - -For more information and advanced setup instructions, check out the [Easy Install Documentation](https://github.com/frappe/bench/blob/develop/docs/easy_install.md). - ### Manual Installation @@ -129,11 +80,6 @@ You'll have to set up the system dependencies required for setting up a Frappe E $ pip install frappe-bench ``` -For more extensive distribution-dependent documentation, check out the following guides: - - - [Hitchhiker's Guide to Installing Frappe on Linux](https://github.com/frappe/frappe/wiki/The-Hitchhiker%27s-Guide-to-Installing-Frappe-on-Linux) - - [Hitchhiker's Guide to Installing Frappe on MacOS](https://github.com/frappe/bench/wiki/Setting-up-a-Mac-for-Frappe-ERPNext-Development) - ## Basic Usage @@ -189,22 +135,6 @@ For more in-depth information on commands and their usage, follow [Commands and If you wish to extend the capabilities of bench with your own custom Frappe Application, you may follow [Adding Custom Bench Commands](https://github.com/frappe/bench/blob/develop/docs/bench_custom_cmd.md). -## Bench Manager - -[Bench Manager](https://github.com/frappe/bench_manager) is a GUI frontend for Bench with the same functionalties. You can install it by executing the following command: - -```sh -$ bench setup manager -``` - - - **Note:** This will create a new site to setup Bench Manager, if you want to set it up on an existing site, run the following commands: - - ```sh - $ bench get-app https://github.com/frappe/bench_manager.git - $ bench --site install-app bench_manager - ``` - - ## Guides - [Configuring HTTPS](https://frappe.io/docs/user/en/bench/guides/configuring-https.html) diff --git a/bench/app.py b/bench/app.py index 0a13d4bd..07eb4799 100755 --- a/bench/app.py +++ b/bench/app.py @@ -198,7 +198,7 @@ class App(AppMeta): @step(title="Archiving App {repo}", success="App {repo} Archived") def remove(self, no_backup: bool = False): - active_app_path = os.path.join("apps", self.name) + active_app_path = os.path.join("apps", self.repo) if no_backup: if not os.path.islink(active_app_path): diff --git a/bench/bench.py b/bench/bench.py index 8586c712..e799838a 100644 --- a/bench/bench.py +++ b/bench/bench.py @@ -147,7 +147,7 @@ class Bench(Base, Validator): if conf.get("developer_mode"): restart_process_manager(bench_path=self.name, web_workers=web) - if supervisor and conf.get("restart_supervisor_on_update"): + if supervisor or conf.get("restart_supervisor_on_update"): restart_supervisor_processes(bench_path=self.name, web_workers=web) if systemd and conf.get("restart_systemd_on_update"): restart_systemd_processes(bench_path=self.name, web_workers=web) diff --git a/bench/config/common_site_config.py b/bench/config/common_site_config.py index d278dedb..df58daf1 100644 --- a/bench/config/common_site_config.py +++ b/bench/config/common_site_config.py @@ -15,6 +15,8 @@ default_config = { "live_reload": True, } +DEFAULT_MAX_REQUESTS = 5000 + def setup_config(bench_path): make_pid_folder(bench_path) @@ -62,6 +64,20 @@ def get_gunicorn_workers(): return {"gunicorn_workers": multiprocessing.cpu_count() * 2 + 1} +def compute_max_requests_jitter(max_requests: int) -> int: + return int(max_requests * 0.1) + + +def get_default_max_requests(worker_count: int): + """Get max requests and jitter config based on number of available workers.""" + + if worker_count <= 1: + # If there's only one worker then random restart can cause spikes in response times and + # can be annoying. Hence not enabled by default. + return 0 + return DEFAULT_MAX_REQUESTS + + def update_config_for_frappe(config, bench_path): ports = make_ports(bench_path) diff --git a/bench/config/supervisor.py b/bench/config/supervisor.py index d38da668..b78c0470 100644 --- a/bench/config/supervisor.py +++ b/bench/config/supervisor.py @@ -8,7 +8,12 @@ import bench from bench.app import use_rq from bench.utils import get_bench_name, which from bench.bench import Bench -from bench.config.common_site_config import update_config, get_gunicorn_workers +from bench.config.common_site_config import ( + update_config, + get_gunicorn_workers, + get_default_max_requests, + compute_max_requests_jitter, +) # imports - third party imports import click @@ -26,6 +31,13 @@ def generate_supervisor_config(bench_path, user=None, yes=False, skip_redis=Fals template = bench.config.env().get_template("supervisor.conf") bench_dir = os.path.abspath(bench_path) + web_worker_count = config.get( + "gunicorn_workers", get_gunicorn_workers()["gunicorn_workers"] + ) + max_requests = config.get( + "gunicorn_max_requests", get_default_max_requests(web_worker_count) + ) + config = template.render( **{ "bench_dir": bench_dir, @@ -39,9 +51,9 @@ def generate_supervisor_config(bench_path, user=None, yes=False, skip_redis=Fals "redis_socketio_config": os.path.join(bench_dir, "config", "redis_socketio.conf"), "redis_queue_config": os.path.join(bench_dir, "config", "redis_queue.conf"), "webserver_port": config.get("webserver_port", 8000), - "gunicorn_workers": config.get( - "gunicorn_workers", get_gunicorn_workers()["gunicorn_workers"] - ), + "gunicorn_workers": web_worker_count, + "gunicorn_max_requests": max_requests, + "gunicorn_max_requests_jitter": compute_max_requests_jitter(max_requests), "bench_name": get_bench_name(bench_path), "background_workers": config.get("background_workers") or 1, "bench_cmd": which("bench"), diff --git a/bench/config/systemd.py b/bench/config/systemd.py index d30edfc9..a19de662 100644 --- a/bench/config/systemd.py +++ b/bench/config/systemd.py @@ -9,7 +9,12 @@ import click import bench from bench.app import use_rq from bench.bench import Bench -from bench.config.common_site_config import get_gunicorn_workers, update_config +from bench.config.common_site_config import ( + get_gunicorn_workers, + update_config, + get_default_max_requests, + compute_max_requests_jitter, +) from bench.utils import exec_cmd, which, get_bench_name @@ -61,6 +66,13 @@ def generate_systemd_config( get_bench_name(bench_path) + "-frappe-long-worker@" + str(i + 1) + ".service" ) + web_worker_count = config.get( + "gunicorn_workers", get_gunicorn_workers()["gunicorn_workers"] + ) + max_requests = config.get( + "gunicorn_max_requests", get_default_max_requests(web_worker_count) + ) + bench_info = { "bench_dir": bench_dir, "sites_dir": os.path.join(bench_dir, "sites"), @@ -73,9 +85,9 @@ def generate_systemd_config( "redis_socketio_config": os.path.join(bench_dir, "config", "redis_socketio.conf"), "redis_queue_config": os.path.join(bench_dir, "config", "redis_queue.conf"), "webserver_port": config.get("webserver_port", 8000), - "gunicorn_workers": config.get( - "gunicorn_workers", get_gunicorn_workers()["gunicorn_workers"] - ), + "gunicorn_workers": web_worker_count, + "gunicorn_max_requests": max_requests, + "gunicorn_max_requests_jitter": compute_max_requests_jitter(max_requests), "bench_name": get_bench_name(bench_path), "worker_target_wants": " ".join(background_workers), "bench_cmd": which("bench"), @@ -93,7 +105,7 @@ def generate_systemd_config( setup_web_config(bench_info, bench_path) setup_redis_config(bench_info, bench_path) - update_config({"restart_systemd_on_update": True}, bench_path=bench_path) + update_config({"restart_systemd_on_update": False}, bench_path=bench_path) update_config({"restart_supervisor_on_update": False}, bench_path=bench_path) diff --git a/bench/config/templates/Procfile b/bench/config/templates/Procfile index d9391087..899d1290 100644 --- a/bench/config/templates/Procfile +++ b/bench/config/templates/Procfile @@ -11,9 +11,7 @@ watch: bench watch {% endif %} {% if use_rq -%} schedule: bench schedule -worker_short: bench worker --queue short 1>> logs/worker.log 2>> logs/worker.error.log -worker_long: bench worker --queue long 1>> logs/worker.log 2>> logs/worker.error.log -worker_default: bench worker --queue default 1>> logs/worker.log 2>> logs/worker.error.log +worker: bench worker 1>> logs/worker.log 2>> logs/worker.error.log {% for worker_name, worker_details in workers.items() %} worker_{{ worker_name }}: bench worker --queue {{ worker_name }} 1>> logs/worker.log 2>> logs/worker.error.log {% endfor %} diff --git a/bench/config/templates/supervisor.conf b/bench/config/templates/supervisor.conf index 085cc2cf..f29c1673 100644 --- a/bench/config/templates/supervisor.conf +++ b/bench/config/templates/supervisor.conf @@ -3,7 +3,7 @@ ; killasgroup=true --> send kill signal to child processes too [program:{{ bench_name }}-frappe-web] -command={{ bench_dir }}/env/bin/gunicorn -b 127.0.0.1:{{ webserver_port }} -w {{ gunicorn_workers }} -t {{ http_timeout }} frappe.app:application --preload +command={{ bench_dir }}/env/bin/gunicorn -b 127.0.0.1:{{ webserver_port }} -w {{ gunicorn_workers }} --max-requests {{ gunicorn_max_requests }} --max-requests-jitter {{ gunicorn_max_requests_jitter }} -t {{ http_timeout }} frappe.app:application --preload priority=4 autostart=true autorestart=true diff --git a/bench/config/templates/systemd/frappe-bench-frappe-web.service b/bench/config/templates/systemd/frappe-bench-frappe-web.service index bb2f0e38..0621e1d3 100644 --- a/bench/config/templates/systemd/frappe-bench-frappe-web.service +++ b/bench/config/templates/systemd/frappe-bench-frappe-web.service @@ -6,7 +6,7 @@ PartOf={{ bench_name }}-web.target User={{ user }} Group={{ user }} Restart=always -ExecStart={{ bench_dir }}/env/bin/gunicorn -b 127.0.0.1:{{ webserver_port }} -w {{ gunicorn_workers }} -t {{ http_timeout }} frappe.app:application --preload +ExecStart={{ bench_dir }}/env/bin/gunicorn -b 127.0.0.1:{{ webserver_port }} -w {{ gunicorn_workers }} -t {{ http_timeout }} --max-requests {{ gunicorn_max_requests }} --max-requests-jitter {{ gunicorn_max_requests_jitter }} frappe.app:application --preload StandardOutput=file:{{ bench_dir }}/logs/web.log StandardError=file:{{ bench_dir }}/logs/web.error.log WorkingDirectory={{ sites_dir }} diff --git a/bench/utils/__init__.py b/bench/utils/__init__.py index ff8f8fcd..b61f686e 100644 --- a/bench/utils/__init__.py +++ b/bench/utils/__init__.py @@ -125,7 +125,10 @@ def check_latest_version(): local_version = Version(VERSION) if pypi_version > local_version: - log(f"A newer version of bench is available: {local_version} → {pypi_version}", stderr=True) + log( + f"A newer version of bench is available: {local_version} → {pypi_version}", + stderr=True, + ) def pause_exec(seconds=10): diff --git a/bench/utils/app.py b/bench/utils/app.py index 790792da..5541b548 100644 --- a/bench/utils/app.py +++ b/bench/utils/app.py @@ -185,7 +185,9 @@ def get_required_deps(org, name, branch, deps="hooks.py"): res = requests.get(url=git_api_url, params=params).json() if "message" in res: - git_url = f"https://raw.githubusercontent.com/{org}/{name}/{params['ref']}/{name}/{deps}" + git_url = ( + f"https://raw.githubusercontent.com/{org}/{name}/{params['ref']}/{name}/{deps}" + ) return requests.get(git_url).text return base64.decodebytes(res["content"].encode()).decode() diff --git a/docs/easy_install.md b/docs/easy_install.md deleted file mode 100644 index 91ee5566..00000000 --- a/docs/easy_install.md +++ /dev/null @@ -1,93 +0,0 @@ -# Easy Install Script - -- This script will install the pre-requisites, install bench and setup an ERPNext site `(site1.local under frappe-bench)` -- Passwords for Frappe Administrator and MariaDB (root) will be asked and saved under `~/passwords.txt` -- MariaDB (root) password may be `password` on a fresh server -- You can then login as **Administrator** with the Administrator password -- The log file is saved under `/tmp/logs/install_bench.log` in case you run into any issues during the install. -- If you find any problems, post them on the forum: [https://discuss.erpnext.com](https://discuss.erpnext.com/tags/installation_problem) under the "Install / Update" category. - ---- - -## What will this script do? - -- Install all the pre-requisites -- Install the command line `bench` (under ~/.bench) -- Create a new bench (a folder that will contain your entire frappe/erpnext setup at ~/frappe-bench) -- Create a new ERPNext site on the bench (site1.local) - ---- - -## Getting started with easy install... - -Open your Terminal and enter: - -#### 0. Setup user & Download the install script - -If you are on a fresh server and logged in as root, at first create a dedicated user for frappe -& equip this user with sudo privileges - -``` - adduser [frappe-user] - usermod -aG sudo [frappe-user] -``` - -*(it is very common to use "frappe" as frappe-username, but this comes with the security flaw of ["frappe" ranking very high](https://www.reddit.com/r/dataisbeautiful/comments/b3sirt/i_deployed_over_a_dozen_cyber_honeypots_all_over/?st=JTJ0SC0Q&sh=76e05240) in as a username challenged in hacking attempts. So, for production sites it is highly recommended to use a custom username harder to guess)* - -*(you can specify the flag --home to specify a directory for your [frappe-user]. Bench will follow the home directory specified by the user's home directory e.g. /data/[frappe-user]/frappe-bench)* - -Switch to `[frappe-user]` (using `su [frappe-user]`) and start the setup - - wget https://raw.githubusercontent.com/frappe/bench/develop/install.py - - -#### 1. Run the install script - - sudo python3 install.py - -*Note: `user` flag to create a user and install using that user (By default, the script will create a user with the username `frappe` if the --user flag is not used)* - -For production or development, append the `--production` or `--develop` flag to the command respectively. - - sudo python3 install.py --production --user [frappe-user] - -or - - sudo python3 install.py --develop - sudo python3 install.py --develop --user [frappe-user] - - sudo python3 install.py --production --user [frappe-user] --container - -*Note: `container` flag to install inside a container (this will prevent the `/proc/sys/vm/swappiness: Read-only` file system error)* - - - python3 install.py --production --version 11 --user [frappe-user] - -use --version flag to install specific version - - python3 install.py --production --version 11 --python python2.7 --user [frappe-user] - -use --python flag to specify virtual environments python version, by default script setup python3 - ---- - -## How do I start ERPNext - -1. For development: Go to your bench folder (`~[frappe-user]/frappe-bench` by default) and start the bench with `bench start` -2. For production: Your process will be setup and managed by `nginx` and `supervisor`. Checkout [Setup Production](https://frappe.io/docs/user/en/bench/guides/setup-production.html) for more information. - ---- - -## An error occured mid installation? - -TLDR; Save the logs! - -1. The easy install script starts multiple processes to install prerequisites, system dependencies, requirements, sets up locales, configuration files, etc. - -2. The script pipes all these process outputs and saves it under `/tmp/log/{easy-install-filename}.log` as prompted by the script in the beginning of the script or/and if something went wrong again. - -3. Retain this log file and share it in case you need help with proceeding with the install. Since, the file's saved under `/tmp` it'll be cleared by the system after a reboot. Be careful to save it elsewhere if needed! - -3. A lot of things can go wrong in setting up the environment due to prior settings, company protocols or even breaking changes in system packages and their dependencies. - -4. Sharing your logfile in any issues opened related to this can help us find solutions to it faster and make the script better! diff --git a/install.py b/install.py deleted file mode 100644 index 54248b9c..00000000 --- a/install.py +++ /dev/null @@ -1,499 +0,0 @@ -#!/usr/bin/env python3 - -from __future__ import print_function - -import os -import sys -import subprocess -import getpass -import json -import multiprocessing -import shutil -import platform -import warnings -import datetime - - -tmp_bench_repo = os.path.join('/', 'tmp', '.bench') -tmp_log_folder = os.path.join('/', 'tmp', 'logs') -execution_timestamp = datetime.datetime.utcnow() -execution_day = "{:%Y-%m-%d}".format(execution_timestamp) -execution_time = "{:%H:%M}".format(execution_timestamp) -log_file_name = "easy-install__{0}__{1}.log".format(execution_day, execution_time.replace(':', '-')) -log_path = os.path.join(tmp_log_folder, log_file_name) -log_stream = sys.stdout -distro_required = not ((sys.version_info.major < 3) or (sys.version_info.major == 3 and sys.version_info.minor < 5)) - - -def log(message, level=0): - levels = { - 0: '\033[94m', # normal - 1: '\033[92m', # success - 2: '\033[91m', # fail - 3: '\033[93m' # warn/suggest - } - start = levels.get(level) or '' - end = '\033[0m' - print(start + message + end) - - -def setup_log_stream(args): - global log_stream - sys.stderr = sys.stdout - - if not args.verbose: - if not os.path.exists(tmp_log_folder): - os.makedirs(tmp_log_folder) - log_stream = open(log_path, 'w') - log("Logs are saved under {0}".format(log_path), level=3) - print("Install script run at {0} on {1}\n\n".format(execution_time, execution_day), file=log_stream) - - -def check_environment(): - needed_environ_vars = ['LANG', 'LC_ALL'] - message = '' - - for var in needed_environ_vars: - if var not in os.environ: - message += "\nexport {0}=C.UTF-8".format(var) - - if message: - log("Bench's CLI needs these to be defined!", level=3) - log("Run the following commands in shell: {0}".format(message), level=2) - sys.exit() - - -def check_system_package_managers(): - if 'Darwin' in os.uname(): - if not shutil.which('brew'): - raise Exception(''' - Please install brew package manager before proceeding with bench setup. Please run following - to install brew package manager on your machine, - - /usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)" - ''') - if 'Linux' in os.uname(): - if not any([shutil.which(x) for x in ['apt-get', 'yum']]): - raise Exception('Cannot find any compatible package manager!') - - -def check_distribution_compatibility(): - dist_name, dist_version = get_distribution_info() - supported_dists = { - 'macos': [10.9, 10.10, 10.11, 10.12], - 'ubuntu': [14, 15, 16, 18, 19, 20], - 'debian': [8, 9, 10], - 'centos': [7] - } - - log("Checking System Compatibility...") - if dist_name in supported_dists: - if float(dist_version) in supported_dists[dist_name]: - log("{0} {1} is compatible!".format(dist_name, dist_version), level=1) - else: - log("{0} {1} is detected".format(dist_name, dist_version), level=1) - log("Install on {0} {1} instead".format(dist_name, supported_dists[dist_name][-1]), level=3) - else: - log("Sorry, the installer doesn't support {0}. Aborting installation!".format(dist_name), level=2) - - -def import_with_install(package): - # copied from https://discuss.erpnext.com/u/nikunj_patel - # https://discuss.erpnext.com/t/easy-install-setup-guide-for-erpnext-installation-on-ubuntu-20-04-lts-with-some-modification-of-course/62375/5 - # need to move to top said v13 for fully python3 era - import importlib - - try: - importlib.import_module(package) - except ImportError: - # caveat : pip3 must be installed - - import pip - - pip.main(['install', package]) - finally: - globals()[package] = importlib.import_module(package) - - -def get_distribution_info(): - # return distribution name and major version - if platform.system() == "Linux": - if distro_required: - current_dist = distro.linux_distribution(full_distribution_name=True) - else: - current_dist = platform.dist() - - return current_dist[0].lower(), current_dist[1].rsplit('.')[0] - - elif platform.system() == "Darwin": - current_dist = platform.mac_ver() - return "macos", current_dist[0].rsplit('.', 1)[0] - - -def run_os_command(command_map): - '''command_map is a dictionary of {'executable': command}. For ex. {'apt-get': 'sudo apt-get install -y python2.7'}''' - success = True - - for executable, commands in command_map.items(): - if shutil.which(executable): - if isinstance(commands, str): - commands = [commands] - - for command in commands: - returncode = subprocess.check_call(command, shell=True, stdout=log_stream, stderr=sys.stderr) - success = success and (returncode == 0) - - return success - - -def install_prerequisites(): - # pre-requisites for bench repo cloning - run_os_command({ - 'apt-get': [ - 'sudo apt-get update', - 'sudo apt-get install -y git build-essential python3-setuptools python3-dev libffi-dev' - ], - 'yum': [ - 'sudo yum groupinstall -y "Development tools"', - 'sudo yum install -y epel-release redhat-lsb-core git python-setuptools python-devel openssl-devel libffi-devel' - ] - }) - - # until psycopg2-binary is available for aarch64 (Arm 64-bit), we'll need libpq and libssl dev packages to build psycopg2 from source - if platform.machine() == 'aarch64': - log("Installing libpq and libssl dev packages to build psycopg2 for aarch64...") - run_os_command({ - 'apt-get': ['sudo apt-get install -y libpq-dev libssl-dev'], - 'yum': ['sudo yum install -y libpq-devel openssl-devel'] - }) - - install_package('curl') - install_package('wget') - install_package('git') - install_package('pip3', 'python3-pip') - - run_os_command({ - 'python3': "sudo -H python3 -m pip install --upgrade pip setuptools-rust" - }) - success = run_os_command({ - 'python3': "sudo -H python3 -m pip install --upgrade setuptools wheel cryptography ansible~=2.8.15" - }) - - if not (success or shutil.which('ansible')): - could_not_install('Ansible') - - -def could_not_install(package): - raise Exception('Could not install {0}. Please install it manually.'.format(package)) - - -def is_sudo_user(): - return os.geteuid() == 0 - - -def install_package(package, package_name=None): - if shutil.which(package): - log("{0} already installed!".format(package), level=1) - else: - log("Installing {0}...".format(package)) - package_name = package_name or package - success = run_os_command({ - 'apt-get': ['sudo apt-get install -y {0}'.format(package_name)], - 'yum': ['sudo yum install -y {0}'.format(package_name)], - 'brew': ['brew install {0}'.format(package_name)] - }) - if success: - log("{0} installed!".format(package), level=1) - return success - could_not_install(package) - - -def install_bench(args): - # clone bench repo - if not args.run_travis: - clone_bench_repo(args) - - if not args.user: - if args.production: - args.user = 'frappe' - - elif 'SUDO_USER' in os.environ: - args.user = os.environ['SUDO_USER'] - - else: - args.user = getpass.getuser() - - if args.user == 'root': - raise Exception('Please run this script as a non-root user with sudo privileges, but without using sudo or pass --user=USER') - - # Python executable - dist_name, dist_version = get_distribution_info() - if dist_name=='centos': - args.python = 'python3.6' - else: - args.python = 'python3' - - # create user if not exists - extra_vars = vars(args) - extra_vars.update(frappe_user=args.user) - - extra_vars.update(user_directory=get_user_home_directory(args.user)) - - if os.path.exists(tmp_bench_repo): - repo_path = tmp_bench_repo - else: - repo_path = os.path.join(os.path.expanduser('~'), 'bench') - - extra_vars.update(repo_path=repo_path) - run_playbook('create_user.yml', extra_vars=extra_vars) - - extra_vars.update(get_passwords(args)) - if args.production: - extra_vars.update(max_worker_connections=multiprocessing.cpu_count() * 1024) - - if args.version <= 10: - frappe_branch = "{0}.x.x".format(args.version) - erpnext_branch = "{0}.x.x".format(args.version) - else: - frappe_branch = "version-{0}".format(args.version) - erpnext_branch = "version-{0}".format(args.version) - - # Allow override of frappe_branch and erpnext_branch, regardless of args.version (which always has a default set) - if args.frappe_branch: - frappe_branch = args.frappe_branch - if args.erpnext_branch: - erpnext_branch = args.erpnext_branch - - extra_vars.update(frappe_branch=frappe_branch) - extra_vars.update(erpnext_branch=erpnext_branch) - - bench_name = 'frappe-bench' if not args.bench_name else args.bench_name - extra_vars.update(bench_name=bench_name) - - # Will install ERPNext production setup by default - if args.without_erpnext: - log("Initializing bench {bench_name}:\n\tFrappe Branch: {frappe_branch}\n\tERPNext will not be installed due to --without-erpnext".format(bench_name=bench_name, frappe_branch=frappe_branch)) - else: - log("Initializing bench {bench_name}:\n\tFrappe Branch: {frappe_branch}\n\tERPNext Branch: {erpnext_branch}".format(bench_name=bench_name, frappe_branch=frappe_branch, erpnext_branch=erpnext_branch)) - run_playbook('site.yml', sudo=True, extra_vars=extra_vars) - - if os.path.exists(tmp_bench_repo): - shutil.rmtree(tmp_bench_repo) - - -def clone_bench_repo(args): - '''Clones the bench repository in the user folder''' - branch = args.bench_branch or 'develop' - repo_url = args.repo_url or 'https://github.com/frappe/bench' - - if os.path.exists(tmp_bench_repo): - log('Not cloning already existing Bench repository at {tmp_bench_repo}'.format(tmp_bench_repo=tmp_bench_repo)) - return 0 - elif args.without_bench_setup: - clone_path = os.path.join(os.path.expanduser('~'), 'bench') - log('--without-bench-setup specified, clone path is: {clone_path}'.format(clone_path=clone_path)) - else: - clone_path = tmp_bench_repo - # Not logging repo_url to avoid accidental credential leak in case credential is embedded in URL - log('Cloning bench repository branch {branch} into {clone_path}'.format(branch=branch, clone_path=clone_path)) - - success = run_os_command( - {'git': 'git clone --quiet {repo_url} {bench_repo} --depth 1 --branch {branch}'.format( - repo_url=repo_url, bench_repo=clone_path, branch=branch)} - ) - - return success - - -def passwords_didnt_match(context=''): - log("{} passwords did not match!".format(context), level=3) - - -def get_passwords(args): - """ - Returns a dict of passwords for further use - and creates passwords.txt in the bench user's home directory - """ - log("Input MySQL and Frappe Administrator passwords:") - ignore_prompt = args.run_travis or args.without_bench_setup - mysql_root_password, admin_password = '', '' - passwords_file_path = os.path.join(os.path.expanduser('~' + args.user), 'passwords.txt') - - if not ignore_prompt: - # set passwords from existing passwords.txt - if os.path.isfile(passwords_file_path): - with open(passwords_file_path, 'r') as f: - passwords = json.load(f) - mysql_root_password, admin_password = passwords['mysql_root_password'], passwords['admin_password'] - - # set passwords from cli args - if args.mysql_root_password: - mysql_root_password = args.mysql_root_password - if args.admin_password: - admin_password = args.admin_password - - # prompt for passwords - pass_set = True - while pass_set: - # mysql root password - if not mysql_root_password: - mysql_root_password = getpass.unix_getpass(prompt='Please enter mysql root password: ') - conf_mysql_passwd = getpass.unix_getpass(prompt='Re-enter mysql root password: ') - - if mysql_root_password != conf_mysql_passwd or mysql_root_password == '': - passwords_didnt_match("MySQL") - mysql_root_password = '' - continue - - # admin password, only needed if we're also creating a site - if not admin_password and not args.without_site: - admin_password = getpass.unix_getpass(prompt='Please enter the default Administrator user password: ') - conf_admin_passswd = getpass.unix_getpass(prompt='Re-enter Administrator password: ') - - if admin_password != conf_admin_passswd or admin_password == '': - passwords_didnt_match("Administrator") - admin_password = '' - continue - elif args.without_site: - log("Not creating a new site due to --without-site") - - pass_set = False - else: - mysql_root_password = admin_password = 'travis' - - passwords = { - 'mysql_root_password': mysql_root_password, - 'admin_password': admin_password - } - - if not ignore_prompt: - with open(passwords_file_path, 'w') as f: - json.dump(passwords, f, indent=1) - - log('Passwords saved at ~/passwords.txt') - - return passwords - - -def get_extra_vars_json(extra_args): - # We need to pass production as extra_vars to the playbook to execute conditionals in the - # playbook. Extra variables can passed as json or key=value pair. Here, we will use JSON. - json_path = os.path.join('/', 'tmp', 'extra_vars.json') - extra_vars = dict(list(extra_args.items())) - - with open(json_path, mode='w') as j: - json.dump(extra_vars, j, indent=1, sort_keys=True) - - return ('@' + json_path) - -def get_user_home_directory(user): - # Return home directory /home/USERNAME or anything else defined as home directory in - # passwd for user. - return os.path.expanduser('~'+user) - - -def run_playbook(playbook_name, sudo=False, extra_vars=None): - args = ['ansible-playbook', '-c', 'local', playbook_name , '-vvvv'] - - if extra_vars: - args.extend(['-e', get_extra_vars_json(extra_vars)]) - - if sudo: - user = extra_vars.get('user') or getpass.getuser() - args.extend(['--become', '--become-user={0}'.format(user)]) - - if os.path.exists(tmp_bench_repo): - cwd = tmp_bench_repo - else: - cwd = os.path.join(os.path.expanduser('~'), 'bench') - - playbooks_locations = [os.path.join(cwd, 'bench', 'playbooks'), os.path.join(cwd, 'playbooks')] - playbooks_folder = [x for x in playbooks_locations if os.path.exists(x)][0] - - success = subprocess.check_call(args, cwd=playbooks_folder, stdout=log_stream, stderr=sys.stderr) - return success - - -def setup_script_requirements(): - if distro_required: - install_package('pip3', 'python3-pip') - import_with_install('distro') - - -def parse_commandline_args(): - import argparse - - parser = argparse.ArgumentParser(description='Frappe Installer') - # Arguments develop and production are mutually exclusive both can't be specified together. - # Hence, we need to create a group for discouraging use of both options at the same time. - args_group = parser.add_mutually_exclusive_group() - - args_group.add_argument('--develop', dest='develop', action='store_true', default=False, help='Install developer setup') - args_group.add_argument('--production', dest='production', action='store_true', default=False, help='Setup Production environment for bench') - parser.add_argument('--site', dest='site', action='store', default='site1.local', help='Specify name for your first ERPNext site') - parser.add_argument('--without-site', dest='without_site', action='store_true', default=False, help='Do not create a new site') - parser.add_argument('--verbose', dest='verbose', action='store_true', default=False, help='Run the script in verbose mode') - parser.add_argument('--user', dest='user', help='Install frappe-bench for this user') - parser.add_argument('--bench-branch', dest='bench_branch', help='Clone a particular branch of bench repository') - parser.add_argument('--repo-url', dest='repo_url', help='Clone bench from the given url') - parser.add_argument('--frappe-repo-url', dest='frappe_repo_url', action='store', default='https://github.com/frappe/frappe', help='Clone frappe from the given url') - parser.add_argument('--frappe-branch', dest='frappe_branch', action='store', help='Clone a particular branch of frappe') - parser.add_argument('--erpnext-repo-url', dest='erpnext_repo_url', action='store', default='https://github.com/frappe/erpnext', help='Clone erpnext from the given url') - parser.add_argument('--erpnext-branch', dest='erpnext_branch', action='store', help='Clone a particular branch of erpnext') - parser.add_argument('--without-erpnext', dest='without_erpnext', action='store_true', default=False, help='Prevent fetching ERPNext') - # direct provision to install versions - parser.add_argument('--version', dest='version', action='store', default=13, type=int, help='Clone particular version of frappe and erpnext') - # To enable testing of script using Travis, this should skip the prompt - parser.add_argument('--run-travis', dest='run_travis', action='store_true', default=False, help=argparse.SUPPRESS) - parser.add_argument('--without-bench-setup', dest='without_bench_setup', action='store_true', default=False, help=argparse.SUPPRESS) - # whether to overwrite an existing bench - parser.add_argument('--overwrite', dest='overwrite', action='store_true', default=False, help='Whether to overwrite an existing bench') - # set passwords - parser.add_argument('--mysql-root-password', dest='mysql_root_password', help='Set mysql root password') - parser.add_argument('--mariadb-version', dest='mariadb_version', default='10.4', help='Specify mariadb version') - parser.add_argument('--admin-password', dest='admin_password', help='Set admin password') - parser.add_argument('--bench-name', dest='bench_name', help='Create bench with specified name. Default name is frappe-bench') - # Python interpreter to be used - parser.add_argument('--python', dest='python', default='python3', help=argparse.SUPPRESS) - # LXC Support - parser.add_argument('--container', dest='container', default=False, action='store_true', help='Use if you\'re creating inside LXC') - - args = parser.parse_args() - - return args - - -if __name__ == '__main__': - if sys.version[0] == '2': - if not os.environ.get('CI'): - if not raw_input("It is recommended to run this script with Python 3\nDo you still wish to continue? [Y/n]: ").lower() == "y": - sys.exit() - - try: - from distutils.spawn import find_executable - except ImportError: - try: - subprocess.check_call('pip install --upgrade setuptools') - except subprocess.CalledProcessError: - print("Install distutils or use Python3 to run the script") - sys.exit(1) - - shutil.which = find_executable - - if not is_sudo_user(): - log("Please run this script as a non-root user with sudo privileges", level=3) - sys.exit() - - args = parse_commandline_args() - - with warnings.catch_warnings(): - warnings.simplefilter("ignore") - setup_log_stream(args) - install_prerequisites() - setup_script_requirements() - check_distribution_compatibility() - check_system_package_managers() - check_environment() - install_bench(args) - - log("Bench + Frappe + ERPNext has been successfully installed!")