Merge branch 'develop' of github.com:frappe/bench into change-project-name

README.md

@@ -1,131 +1,236 @@
 <div align="center">
 	<img src="https://github.com/frappe/design/raw/master/logos/png/bench-logo.png" height="128">
-	<h2>bench</h2>
+	<h2>Bench</h2>
 </div>
 
-bench is a command-line utility that helps you to install apps, manage multiple sites and update Frappe / ERPNext apps on */nix (macOS, Ubuntu, Debian, CentOS, etc) for development and production.
+Bench is a command-line utility that helps you to install, update, and manage multiple sites for Frappe/ERPNext applications on [*nix systems](https://en.wikipedia.org/wiki/Unix-like) for development and production.
 
+## Table of Contents
 
-> **Note**: If you are looking for easier ways to get started and evaluate ERPNext, [download the Virtual Machine](https://erpnext.com/download) or take [a free trial on erpnext.com](https://erpnext.com/pricing).
-
----
-
-# Table of Contents
-
- - [bench CLI](#bench-cli)
-	- [Usage](#usage)
  - [Installation](#installation)
-	- [Custom Bench commands](#custom-bench-commands)
+	- [Docker Installation](#docker-installation)
+		- [Development Setup](#docker-installation-for-development)
+		- [Production Setup](#docker-installation-for-production)
 	- [Easy Install Script](#easy-install-script)
- - [Release Bench](#release-bench)
+	- [Manual Installation](#manual-installation)
+ - [Usage](#usage)
+ - [Custom Bench commands](#custom-bench-commands)
+ - [Bench Manager](#bench-manager)
  - [Guides](#guides)
  - [Resources](#resources)
  - [License](#license)
----
 
-# bench CLI
-
-Bench is a command line tool that helps you install, setup, manage multiple sites and apps based on Frappe Framework.
-
----
-
-## Usage
-
-* Create a new bench
-
-		bench init [bench-name]
-
-* Add a site under current bench
-
-		bench new-site [site-name]
-
-	**Optional**: If the database for the site does not reside on localhost or listens on a custom port, you can use the flags `--db-host` to set a custom host and/or `--db-port` to set a custom port.
-
-		bench new-site [site-name] --db-host [custom-db-host-ip] --db-port [custom-db-port]
-
-* Add apps to bench
-
-		bench get-app [app-name] [app-link]
-
-* Install apps on a particular site
-
-		bench --site [site-name] install-app [app-name]
-
-* Start bench (only for development)
-
-		bench start
-
-* Show bench help
-
-		bench --help
-
-_Note:_ Apart from `bench init`, all other bench commands have to be run having the respective bench directory as the working directory. _(`bench update` may also be run, but it's behaviour is covered in depth in the docs)_
-
-For more in depth information on commands and usage follow [here](https://github.com/frappe/bench/blob/master/docs/commands_and_usage.md). As for a consolidated list of bench commands, go through [this page](https://github.com/frappe/bench/blob/master/docs/bench_usage.md).
-
----
 
 ## Installation
 
-To do this install, you must have basic information on how Linux works and should be able to use the command-line. bench will also create nginx and supervisor config files, setup backups and much more. If you are using on a VPS make sure it has >= 1Gb of RAM or has swap setup properly.
+A typical bench setup provides two types of environments &mdash; Development and Production.
 
-	git clone https://github.com/frappe/bench ~/.bench
+The setup for each of these installations can be achieved in multiple ways:
-	pip3 install --user -e ~/.bench
 
-As bench is a python application, its installation really depends on `python` + `pip` + `git`. The Frappe Framework, however has various other system dependencies like `nodejs`, `yarn`, `redis` and a database system like `mariadb` or `postgres`. Go through the [installation requirements](https://github.com/frappe/bench/blob/master/docs/installation.md) for an updated list.
+ - [Docker Installation](#docker-installation)
+ - [Easy Install Script](#easy-install-script)
+ - [Manual Installation](#manual-installation)
 
-If you have questions, please ask them on the [forum](https://discuss.erpnext.com/c/bench) under the "Install / Update" category.
+We recommend using either the Docker Installation or the Easy Install Script to setup a Production Environment. For Development, you may choose either of the three methods to setup an instance.
+
+Otherwise, if you are looking to evaluate ERPNext, you can also download the [Virtual Machine Image](https://erpnext.com/download) or register for [a free trial on erpnext.com](https://erpnext.com/pricing).
+
+
+### Docker Installation
+
+A Frappe/ERPNext instance can be setup and replicated easily using [Docker](https://docker.com). The officially supported Docker installation can be used to setup either of both Development and Production environments.
+
+To setup either of the environments, you will need to clone the official docker repository:
+
+```sh
+$ git clone https://github.com/frappe/frappe_docker.git
+$ cd frappe_docker
+```
+
+A quick setup guide for both the envionments can be found below. For more details, check out the [Frappe/ERPNext Docker Repository](https://github.com/frappe/frappe_docker).
+
+#### Docker Installation for Development
+
+To setup a development environment for Docker, follow the [Frappe/ERPNext Docker for Development Guide](https://github.com/frappe/frappe_docker/blob/develop/development/README.md).
+
+#### Docker Installation for Production
+
+Copy the `env-example` file to `.env`
+
+```sh
+$ cp installation/env-example installation/.env
+```
+
+Make a directory for handling sites:
+
+```sh
+$ mkdir installation/sites
+```
+
+Optionally, you may also setup an [NGINX Proxy for SSL Certificates](https://github.com/evertramos/docker-compose-letsencrypt-nginx-proxy-companion) with auto-renewal for your Production instance. We recommend this for instances being accessed over the internet. For this to work, the DNS needs to be configured correctly so that [LetsEncrypt](https://letsencrypt.org) can verify the domain. To setup the proxy, run the following commands:
+
+```sh
+$ git clone https://github.com/evertramos/docker-compose-letsencrypt-nginx-proxy-companion.git
+$ cd docker-compose-letsencrypt-nginx-proxy-companion
+$ cp .env.sample .env
+$ ./start.sh
+```
+
+To get the Production instance running, run the following command:
+
+```sh
+$ docker-compose \
+    --project-name <project-name> \
+    -f installation/docker-compose-common.yml \
+    -f installation/docker-compose-erpnext.yml \
+    -f installation/docker-compose-networks.yml \
+    --project-directory installation up -d
+```
+
+Make sure to replace `<project-name>` with whatever you wish to call it. This should get the instance running through docker. Now, to create a new site on the instance you may run:
+
+```sh
+docker exec -it \
+    -e "SITE_NAME=$SITE_NAME" \
+    -e "DB_ROOT_USER=$DB_ROOT_USER" \
+    -e "MYSQL_ROOT_PASSWORD=$MYSQL_ROOT_PASSWORD" \
+    -e "ADMIN_PASSWORD=$ADMIN_PASSWORD" \
+    -e "INSTALL_APPS=erpnext" \ # optional, if you want to install any other apps
+    <project-name>_erpnext-python_1 docker-entrypoint.sh new
+```
+
+Once this is done, you may access the instance at `$SITE_NAME`.
+
+**Note:** The Production setup does not contain, require, or use bench. For a list of substitute commands, check out the [Frappe/ERPNext Docker Site Operations](https://github.com/frappe/frappe_docker/#site-operations).
+
+
+### Easy Install Script
+
+The Easy Install script should get you going with a Frappe/ERPNext setup with minimal manual intervention and effort. Since there are a lot of configurations being automatically setup, we recommend executing this script on a fresh server.
+
+**Note:** This script works only on GNU/Linux based server distributions, and has been designed and tested to work on Ubuntu 16.04+, CentOS 7+, and Debian-based systems.
+
+#### Prerequisites
+
+You need to install the following packages for the script to run:
+
+ - ##### Ubuntu and Debian-based Distributions:
+
+	```sh
+	$ apt install python3-minimal build-essential python3-setuptools
+	```
+
+ - ##### CentOS and other RPM Distributions:
+
+	```sh
+	$ dnf groupinstall "Development Tools"
+	$ dnf install python3
+	```
+
+#### Setup
+
+Download the Easy Install script and execute it:
+
+```sh
+$ wget https://raw.githubusercontent.com/frappe/bench/master/playbooks/install.py
+$ python3 install.py --production
+```
+
+The script should then prompt you for the MySQL root password and an Administrator password for the Frappe/ERPNext instance, which will then be saved under `$HOME/passwords.txt` of the user used to setup the instance. This script will then install the required stack, setup bench and a default ERPNext instance.
+
+When the setup is complete, you will be able to access the system at `http://<your-server-ip>`, wherein you can use the administrator password to login.
+
+#### Troubleshooting
+
+In case the setup fails, the log file is saved under `/tmp/logs/install_bench.log`. You may then: 
+
+ - Create an Issue in this repository with the log file attached.
+ - Search for an existing issue or post the log file on the [Frappe/ERPNext Discuss Forum](https://discuss.erpnext.com/c/bench) with the tag `installation_problem` under "Install/Update" category.
+
+For more information and advanced setup instructions, check out the [Easy Install Documentation](https://github.com/frappe/bench/blob/master/docs/easy_install.md).
+
+
+### Manual Installation
+
+Although not recommended, some might want to manually setup a bench instance locally for development. To quickly get started on installing bench the hard way, you can follow [Installing Bench and Frappe](https://frappe.io/docs/user/en/installation).
+
+For more extensive distribution-dependent documentation, check out the following guides:
+
+ - [Hitchhiker's Guide to Installing Frappe on Linux](https://github.com/frappe/frappe/wiki/The-Hitchhiker%27s-Guide-to-Installing-Frappe-on-Linux)
+ - [Hitchhiker's Guide to Installing Frappe on MacOS](https://github.com/frappe/frappe/wiki/The-Hitchhiker%27s-Guide-to-Installing-Frappe-on-Mac-OS-X)
+
+
+## Basic Usage
+
+**Note:** Apart from `bench init`, all other bench commands are expected to be run in the respective bench directory.
+
+ * Create a new bench:
+
+	```sh
+	$ bench init [bench-name]
+	```
+
+ * Add a site under current bench:
+
+	```sh
+	$ bench new-site [site-name]
+	```
+	- **Optional**: If the database for the site does not reside on localhost or listens on a custom port, you can use the flags `--db-host` to set a custom host and/or `--db-port` to set a custom port.
+
+		```sh
+		$ bench new-site [site-name] --db-host [custom-db-host-ip] --db-port [custom-db-port]
+		```
+
+ * Download and add applications to bench:
+
+	```sh
+	$ bench get-app [app-name] [app-link]
+	```
+
+ * Install apps on a particular site
+
+	```sh
+	$ bench --site [site-name] install-app [app-name]
+	```
+
+ * Start bench (only for development)
+
+	```sh
+	$ bench start
+	```
+
+ * Show bench help:
+
+	```sh
+	$ bench --help
+	```
+
+
+For more in-depth information on commands and their usage, follow [Commands and Usage](https://github.com/frappe/bench/blob/master/docs/commands_and_usage.md). As for a consolidated list of bench commands, check out [Bench Usage](https://github.com/frappe/bench/blob/master/docs/bench_usage.md).
 
----
 
 ## Custom Bench Commands
 
-Want to utilize a bench command you've added in your custom Frappe application? [This](https://github.com/frappe/bench/blob/master/docs/bench_custom_cmd.md) guide might be of some help.
+If you wish to extend the capabilities of bench with your own custom Frappe Application, you may follow [Adding Custom Bench Commands](https://github.com/frappe/bench/blob/master/docs/bench_custom_cmd.md).
 
----
 
-# Easy Install Script
+## Bench Manager
 
-- This is an opinionated setup so it is best to setup on a blank server.
+[Bench Manager](https://github.com/frappe/bench_manager) is a GUI frontend for Bench with the same functionalties. You can install it by executing the following command:
-- Works on Ubuntu 16.04+, CentOS 7+, Debian 8+
-- You may have to install Python 3 and other essentials by running `apt-get install python3-minimal build-essential python3-setuptools`
-- This script will install the pre-requisites, install bench and setup an ERPNext site `(site1.local under frappe-bench)`
-- Passwords for Frappe Administrator and MariaDB (root) will be asked and saved under `~/passwoords.txt`
-- MariaDB (root) password may be `password` on a fresh server
-- You can then login as **Administrator** with the Administrator password
-- The log file is saved under `/tmp/logs/install_bench.log` in case you run into any issues during the install.
-- If you find any problems, post them on the forum: [https://discuss.erpnext.com](https://discuss.erpnext.com/c/bench) with the `installation_problem` under "Install / Update" category.
 
-		wget https://raw.githubusercontent.com/frappe/bench/master/playbooks/install.py
+```sh
-		python3 install.py --production
+$ bench setup manager
+```
 
-Follow [Easy Install Docs](https://github.com/frappe/bench/blob/master/docs/easy_install.md) for more information.
+ - **Note:** This will create a new site to setup Bench Manager, if you want to set it up on an existing site, run the following commands:
 
----
+	```sh
+	$ bench get-app https://github.com/frappe/bench_manager.git
+	$ bench --site <sitename> install-app bench_manager
+	```
 
-# Release Bench
 
-Releases can be made for [Frappe](https://github.com/frappe/frappe) apps using bench. More information about this process can be found [here](https://github.com/frappe/bench/blob/master/docs/releasing_frappe_apps.md).
+## Guides
-
----
-
-# Bench Manager (GUI for Bench)
-
-[Bench Manager](https://github.com/frappe/bench_manager) is a graphical user interface to emulate the functionalities of Frappe Bench. Like the command line utility it helps you install apps, manage multiple sites, update apps and much more. Install just by executing the following command in the respective bench directory.
-
-		bench setup manager
-
----
-
-# Docker
-
-- For official images and resources [Frappe Docker](https://github.com/frappe/frappe_docker)
-- Production Installation [README](https://github.com/frappe/frappe_docker/blob/develop/README.md)
-- Developer Setup [README](https://github.com/frappe/frappe_docker/blob/develop/development/README.md)
-
----
-
-# Guides
 
 - [Configuring HTTPS](https://frappe.io/docs/user/en/bench/guides/configuring-https.html)
 - [Using Let's Encrypt to setup HTTPS](https://frappe.io/docs/user/en/bench/guides/lets-encrypt-ssl-setup.html)
@@ -136,16 +241,18 @@ Releases can be made for [Frappe](https://github.com/frappe/frappe) apps using b
 - [Setup Multitenancy](https://frappe.io/docs/user/en/bench/guides/setup-multitenancy.html)
 - [Stopping Production](https://github.com/frappe/bench/wiki/Stopping-Production-and-starting-Development)
 
----
+For an exhaustive list of guides, check out [Bench Guides](https://frappe.io/docs/user/en/bench/guides).
 
-# Resources
 
-- [Background Services](https://frappe.io/docs/user/en/bench/resources/background-services.html)
+## Resources
+
 - [Bench Commands Cheat Sheet](https://frappe.io/docs/user/en/bench/resources/bench-commands-cheatsheet.html)
+- [Background Services](https://frappe.io/docs/user/en/bench/resources/background-services.html)
 - [Bench Procfile](https://frappe.io/docs/user/en/bench/resources/bench-procfile.html)
 
----
+For an exhaustive list of resources, check out [Bench Resources](https://frappe.io/docs/user/en/bench/resources).
 
-# License
 
-bench is licensed under [GNU GPLv3](LICENSE)
+## License
+
+This repository has been released under the [GNU GPLv3 License](LICENSE).

bench/__init__.py

@@ -1,7 +1,6 @@
 VERSION = "5.0.0"
 PROJECT_NAME = "frappe-bench"
 FRAPPE_VERSION = None
-__version__ = VERSION
 
 
 def set_frappe_version(bench_path='.'):

bench/app.py

@@ -111,6 +111,9 @@ def get_app(git_url, branch=None, bench_path='.', skip_assets=False, verbose=Fal
 						if git_url == data['name']:
 							git_url = 'https://github.com/{org}/{app}'.format(org=org, app=git_url)
 							break
+				else:
+					bench.utils.log("App {app} not found".format(app=git_url), level=2)
+					sys.exit(1)
 
 		# Gets repo name from URL
 		repo_name = git_url.rsplit('/', 1)[1].rsplit('.', 1)[0]
@@ -177,8 +180,8 @@ def install_app(app, bench_path=".", verbose=False, no_cache=False, postprocess=
 	app_path = os.path.join(bench_path, "apps", app)
 	cache_flag = "--no-cache-dir" if no_cache else ""
 
-	exec_cmd("{pip} install {quiet} -U -e {app} {no_cache}".format(pip=pip_path,
+	exec_cmd("{pip} install {quiet} -U -e {app} {no_cache}".format(pip=pip_path, quiet=quiet_flag, app=app_path, no_cache=cache_flag))
-									quiet=quiet_flag, app=app_path, no_cache=cache_flag))
+	exec_cmd("yarn install", cwd=app_path)
 	add_to_appstxt(app, bench_path=bench_path)
 
 	if postprocess:
@@ -218,13 +221,14 @@ def remove_app(app, bench_path='.'):
 	if get_config(bench_path).get('restart_systemd_on_update'):
 		restart_systemd_processes(bench_path=bench_path)
 
-def pull_all_apps(bench_path='.', reset=False):
+def pull_apps(apps=None, bench_path='.', reset=False):
 	'''Check all apps if there no local changes, pull'''
 	rebase = '--rebase' if get_config(bench_path).get('rebase_on_pull') else ''
 
+	apps = apps or get_apps(bench_path=bench_path)
 	# chech for local changes
 	if not reset:
-		for app in get_apps(bench_path=bench_path):
+		for app in apps:
 			excluded_apps = get_excluded_apps()
 			if app in excluded_apps:
 				print("Skipping reset for app {}".format(app))
@@ -248,7 +252,7 @@ Here are your choices:
 					sys.exit(1)
 
 	excluded_apps = get_excluded_apps()
-	for app in get_apps(bench_path=bench_path):
+	for app in apps:
 		if app in excluded_apps:
 			print("Skipping pull for app {}".format(app))
 			continue

bench/cli.py

@@ -1,9 +1,9 @@
 # imports - standard imports
+import atexit
 import json
 import logging
 import os
 import pwd
-import subprocess
 import sys
 
 # imports - third party imports
@@ -14,8 +14,7 @@ import bench
 from bench.app import get_apps
 from bench.commands import bench_command
 from bench.config.common_site_config import get_config
-from bench.utils import PatchError, drop_privileges, find_parent_bench, get_cmd_output, get_env_cmd, get_frappe, is_dist_editable, is_root, log
+from bench.utils import PatchError, bench_cache_file, check_latest_version, drop_privileges, find_parent_bench, generate_command_cache, get_cmd_output, get_env_cmd, get_frappe, is_bench_directory, is_dist_editable, is_root, log
-
 
 logger = logging.getLogger(bench.PROJECT_NAME)
 from_command_line = False
@@ -31,44 +30,51 @@ def cli():
 	change_dir()
 	change_uid()
 
-	if is_dist_editable(bench.PROJECT_NAME):
+	if is_dist_editable(bench.PROJECT_NAME) and len(sys.argv) > 1 and sys.argv[1] != "src":
 		log("bench is installed in editable mode!\n\nThis is not the recommended mode of installation for production. Instead, install the package from PyPI with: `pip install frappe-bench`\n", level=3)
 
+	if not is_bench_directory() and not cmd_requires_root() and len(sys.argv) > 1 and sys.argv[1] not in ("init", "find", "src"):
+		log("Command not being executed in bench directory", level=3)
+
 	if len(sys.argv) > 2 and sys.argv[1] == "frappe":
 		return old_frappe_cli()
 
-	elif len(sys.argv) > 1 and sys.argv[1] in get_frappe_commands():
+	elif len(sys.argv) > 1:
+		if sys.argv[1] in get_frappe_commands() + ["--site", "--verbose", "--force", "--profile"]:
 			return frappe_cmd()
 
-	elif len(sys.argv) > 1 and sys.argv[1] in ("--site", "--verbose", "--force", "--profile"):
+		elif sys.argv[1] == "--help":
-		return frappe_cmd()
-
-	elif len(sys.argv) > 1 and sys.argv[1]=="--help":
 			print(click.Context(bench_command).get_help())
 			print(get_frappe_help())
 			return
 
-	elif len(sys.argv) > 1 and sys.argv[1] in get_apps():
+		elif sys.argv[1] in get_apps():
 			return app_cmd()
 
-	else:
+	if not (len(sys.argv) > 1 and sys.argv[1] == "src"):
+		atexit.register(check_latest_version)
+
 	try:
 		bench_command()
 	except PatchError:
 		sys.exit(1)
 
+
 def check_uid():
 	if cmd_requires_root() and not is_root():
 		log('superuser privileges required for this command', level=3)
 		sys.exit(1)
 
+
 def cmd_requires_root():
 	if len(sys.argv) > 2 and sys.argv[2] in ('production', 'sudoers', 'supervisor', 'lets-encrypt', 'fonts',
 		'print', 'firewall', 'ssh-port', 'role', 'fail2ban', 'wildcard-ssl'):
 		return True
-	if len(sys.argv) >= 2 and sys.argv[1] in ('patch', 'renew-lets-encrypt', 'disable-production',
+	if len(sys.argv) >= 2 and sys.argv[1] in ('patch', 'renew-lets-encrypt', 'disable-production'):
-		'install'):
 		return True
+	if len(sys.argv) > 2 and sys.argv[1] in ('install'):
+		return True
+
 
 def change_dir():
 	if os.path.exists('config.json') or "init" in sys.argv:
@@ -80,6 +86,7 @@ def change_dir():
 		if os.path.exists(dir_path):
 			os.chdir(dir_path)
 
+
 def change_uid():
 	if is_root() and not cmd_requires_root():
 		frappe_user = get_config(".").get('frappe_user')
@@ -90,35 +97,37 @@ def change_uid():
 			log(change_uid_msg, level=3)
 			sys.exit(1)
 
+
 def old_frappe_cli(bench_path='.'):
 	f = get_frappe(bench_path=bench_path)
 	os.chdir(os.path.join(bench_path, 'sites'))
 	os.execv(f, [f] + sys.argv[2:])
 
+
 def app_cmd(bench_path='.'):
 	f = get_env_cmd('python', bench_path=bench_path)
 	os.chdir(os.path.join(bench_path, 'sites'))
 	os.execv(f, [f] + ['-m', 'frappe.utils.bench_helper'] + sys.argv[1:])
 
+
 def frappe_cmd(bench_path='.'):
 	f = get_env_cmd('python', bench_path=bench_path)
 	os.chdir(os.path.join(bench_path, 'sites'))
 	os.execv(f, [f] + ['-m', 'frappe.utils.bench_helper', 'frappe'] + sys.argv[1:])
 
-def get_frappe_commands(bench_path='.'):
+
-	python = get_env_cmd('python', bench_path=bench_path)
+def get_frappe_commands():
-	sites_path = os.path.join(bench_path, 'sites')
+	if not is_bench_directory():
-	if not os.path.exists(sites_path):
-		log("Command not being executed in bench directory", level=3)
-		return []
-	try:
-		output = get_cmd_output("{python} -m frappe.utils.bench_helper get-frappe-commands".format(python=python), cwd=sites_path)
-		return json.loads(output)
-	except subprocess.CalledProcessError as e:
-		if hasattr(e, "stderr"):
-			print(e.stderr.decode('utf-8'))
 		return []
 
+	if os.path.exists(bench_cache_file):
+		command_dump = open(bench_cache_file, 'r').read() or '[]'
+		return json.loads(command_dump)
+
+	else:
+		return generate_command_cache()
+
+
 def get_frappe_help(bench_path='.'):
 	python = get_env_cmd('python', bench_path=bench_path)
 	sites_path = os.path.join(bench_path, 'sites')
@@ -128,6 +137,7 @@ def get_frappe_help(bench_path='.'):
 	except:
 		return ""
 
+
 def change_working_directory():
 	"""Allows bench commands to be run from anywhere inside a bench directory"""
 	cur_dir = os.path.abspath(".")

bench/commands/__init__.py

@@ -7,7 +7,7 @@ def print_bench_version(ctx, param, value):
 		return
 
 	import bench
-	click.echo(bench.__version__)
+	click.echo(bench.VERSION)
 	ctx.exit()
 
 @click.group()
@@ -40,7 +40,8 @@ bench_command.add_command(switch_to_develop)
 
 from bench.commands.utils import (start, restart, set_nginx_port, set_ssl_certificate, set_ssl_certificate_key, set_url_root,
 	set_mariadb_host, set_default_site, download_translations, backup_site, backup_all_sites, release, renew_lets_encrypt,
-	disable_production, bench_src, prepare_beta_release, set_redis_cache_host, set_redis_queue_host, set_redis_socketio_host, find_benches, migrate_env)
+	disable_production, bench_src, prepare_beta_release, set_redis_cache_host, set_redis_queue_host, set_redis_socketio_host, find_benches, migrate_env,
+	generate_command_cache, clear_command_cache)
 bench_command.add_command(start)
 bench_command.add_command(restart)
 bench_command.add_command(set_nginx_port)
@@ -62,7 +63,8 @@ bench_command.add_command(bench_src)
 bench_command.add_command(prepare_beta_release)
 bench_command.add_command(find_benches)
 bench_command.add_command(migrate_env)
-
+bench_command.add_command(generate_command_cache)
+bench_command.add_command(clear_command_cache)
 
 from bench.commands.setup import setup
 bench_command.add_command(setup)

bench/commands/update.py

@@ -1,17 +1,14 @@
-# imports - standard imports
-import os
-
 # imports - third party imports
 import click
-from six.moves import reload_module
 
 # imports - module imports
-from bench.app import pull_all_apps
+from bench.app import pull_apps
 from bench.utils import post_upgrade, patch_sites, build_assets
 
 
 @click.command('update', help="Updates bench tool and if executed in a bench directory, without any flags will backup, pull, setup requirements, build, run patches and restart bench. Using specific flags will only do certain tasks instead of all")
 @click.option('--pull', is_flag=True, help="Pull updates for all the apps in bench")
+@click.option('--apps', type=str)
 @click.option('--patch', is_flag=True, help="Run migrations for all sites in the bench")
 @click.option('--build', is_flag=True, help="Build JS and CSS assets for the bench")
 @click.option('--requirements', is_flag=True, help="Update requirements. If run alone, equivalent to `bench setup requirements`")
@@ -20,15 +17,15 @@ from bench.utils import post_upgrade, patch_sites, build_assets
 @click.option('--no-backup', is_flag=True, help="If this flag is set, sites won't be backed up prior to updates. Note: This is not recommended in production.")
 @click.option('--force', is_flag=True, help="Forces major version upgrades")
 @click.option('--reset', is_flag=True, help="Hard resets git branch's to their new states overriding any changes and overriding rebase on pull")
-def update(pull, patch, build, requirements, restart_supervisor, restart_systemd, no_backup, force, reset):
+def update(pull, apps, patch, build, requirements, restart_supervisor, restart_systemd, no_backup, force, reset):
 	from bench.utils import update
-	update(pull=pull, patch=patch, build=build, requirements=requirements, restart_supervisor=restart_supervisor, restart_systemd=restart_systemd, backup=not no_backup, force=force, reset=reset)
+	update(pull=pull, apps=apps, patch=patch, build=build, requirements=requirements, restart_supervisor=restart_supervisor, restart_systemd=restart_systemd, backup=not no_backup, force=force, reset=reset)
 
 
 @click.command('retry-upgrade', help="Retry a failed upgrade")
 @click.option('--version', default=5)
 def retry_upgrade(version):
-	pull_all_apps()
+	pull_apps()
 	patch_sites()
 	build_assets()
 	post_upgrade(version-1, version)

bench/commands/utils.py

@@ -180,3 +180,15 @@ def find_benches(location):
 def migrate_env(python, backup=True):
 	from bench.utils import migrate_env
 	migrate_env(python=python, backup=backup)
+
+
+@click.command('generate-command-cache', help="Caches Frappe Framework commands")
+def generate_command_cache(bench_path='.'):
+	from bench.utils import generate_command_cache
+	return generate_command_cache(bench_path=bench_path)
+
+
+@click.command('clear-command-cache', help="Clears Frappe Framework cached commands")
+def clear_command_cache(bench_path='.'):
+	from bench.utils import clear_command_cache
+	return clear_command_cache(bench_path=bench_path)

bench/utils.py

@@ -1,3 +1,6 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+
 # imports - standard imports
 import errno
 import glob
@@ -20,7 +23,9 @@ from distutils.spawn import find_executable
 
 # imports - third party imports
 import click
+from crontab import CronTab
 import requests
+from semantic_version import Version
 from six import iteritems
 from six.moves.urllib.parse import urlparse
 
@@ -35,7 +40,7 @@ class CommandFailedError(Exception):
 	pass
 
 logger = logging.getLogger(__name__)
-
+bench_cache_file = '.bench.cmd'
 folders_in_bench = ('apps', 'sites', 'config', 'logs', 'config/pids')
 sudoers_file = '/etc/sudoers.d/frappe'
 
@@ -61,7 +66,7 @@ def is_bench_directory(directory=os.path.curdir):
 
 def log(message, level=0):
 	levels = {
-		0: color.blue + 'LOG',			# normal
+		0: color.blue + 'INFO',			# normal
 		1: color.green + 'SUCCESS',		# success
 		2: color.red + 'ERROR',			# fail
 		3: color.yellow + 'WARN'		# warn/suggest
@@ -80,6 +85,23 @@ def safe_decode(string, encoding = 'utf-8'):
 	return string
 
 
+def check_latest_version():
+	try:
+		pypi_request = requests.get("https://pypi.org/pypi/frappe-bench/json")
+	except Exception:
+		# Exceptions thrown are defined in requests.exceptions
+		# ignore checking on all Exceptions
+		return
+
+	if pypi_request.status_code == 200:
+		pypi_version_str = pypi_request.json().get('info').get('version')
+		pypi_version = Version(pypi_version_str)
+		local_version = Version(bench.VERSION)
+
+		if pypi_version > local_version:
+			log("A newer version of bench is available: {0} → {1}".format(local_version, pypi_version))
+
+
 def get_frappe(bench_path='.'):
 	frappe = get_env_cmd('frappe', bench_path=bench_path)
 	if not os.path.exists(frappe):
@@ -155,17 +177,22 @@ def init(path, apps_path=None, no_procfile=False, no_backups=False,
 	copy_patches_txt(path)
 
 
-def update(pull=False, patch=False, build=False, requirements=False, backup=True, force=False, reset=False,
+def update(pull=False, apps=None, patch=False, build=False, requirements=False, backup=True, force=False, reset=False,
 	restart_supervisor=False, restart_systemd=False):
 	"""command: bench update"""
 	from bench import patches
-	from bench.app import is_version_upgrade, pull_all_apps, validate_branch
+	from bench.app import is_version_upgrade, pull_apps, validate_branch
 	from bench.config.common_site_config import get_config, update_config
 
 	bench_path = os.path.abspath(".")
 	patches.run(bench_path=bench_path)
 	conf = get_config(bench_path)
 
+	if apps and not pull:
+		apps = []
+
+	clear_command_cache(bench_path='.')
+
 	if conf.get('release_bench'):
 		print('Release bench detected, cannot update!')
 		sys.exit(1)
@@ -193,8 +220,11 @@ def update(pull=False, patch=False, build=False, requirements=False, backup=True
 		print('Backing up sites...')
 		backup_all_sites(bench_path=bench_path)
 
+	if apps:
+		apps = [app.strip() for app in re.split(",| ", apps) if app]
+
 	if pull:
-		pull_all_apps(bench_path=bench_path, reset=reset)
+		pull_apps(apps=apps, bench_path=bench_path, reset=reset)
 
 	if requirements:
 		update_requirements(bench_path=bench_path)
@@ -219,7 +249,7 @@ def update(pull=False, patch=False, build=False, requirements=False, backup=True
 	conf.update({ "maintenance_mode": 0, "pause_scheduler": 0 })
 	update_config(conf, bench_path=bench_path)
 
-	print("_" * 80 + "\nBench: Deployment tool for Frappe and Frappe Applications (https://frappe.io/bench).\nOpen source depends on your contributions, so please contribute bug reports, patches, fixes or cash and be a part of the community")
+	print("_" * 80 + "\nBench: Deployment tool for Frappe and Frappe Applications (https://frappe.io/bench).\nOpen source depends on your contributions, so do give back by submitting bug reports, patches and fixes and be a part of the community :)")
 
 
 def copy_patches_txt(bench_path):
@@ -340,42 +370,27 @@ def get_sites(bench_path='.'):
 	return sites
 
 
-def get_bench_dir(bench_path='.'):
-	return os.path.abspath(bench_path)
-
-
 def setup_backups(bench_path='.'):
+	from bench.config.common_site_config import get_config
 	logger.info('setting up backups')
-	bench_dir = get_bench_dir(bench_path=bench_path)
+
+	bench_dir = os.path.abspath(bench_path)
+	user = get_config(bench_path=bench_dir).get('frappe_user')
+	logfile = os.path.join(bench_dir, 'logs', 'backup.log')
 	bench.set_frappe_version(bench_path=bench_path)
+	system_crontab = CronTab(user=user)
 
 	if bench.FRAPPE_VERSION == 4:
 		backup_command = "cd {sites_dir} && {frappe} --backup all".format(frappe=get_frappe(bench_path=bench_path),)
 	else:
-		backup_command = "cd {bench_dir} && {bench} --site all backup".format(bench_dir=bench_dir, bench=sys.argv[0])
+		backup_command = "cd {bench_dir} && {bench} --verbose --site all backup".format(bench_dir=bench_dir, bench=sys.argv[0])
 
-	add_to_crontab('0 */6 * * *  {backup_command} >> {logfile} 2>&1'.format(backup_command=backup_command,
+	job_command = "{backup_command} >> {logfile} 2>&1".format(backup_command=backup_command, logfile=logfile)
-		logfile=os.path.join(get_bench_dir(bench_path=bench_path), 'logs', 'backup.log')))
 
-
+	if job_command not in str(system_crontab):
-def add_to_crontab(line):
+		job = system_crontab.new(command=job_command, comment="bench auto backups set for every 6 hours")
-	current_crontab = read_crontab()
+		job.hour.every(6)
-	line = str.encode(line)
+		system_crontab.write()
-	if not line in current_crontab:
-		cmd = ["crontab"]
-		if platform.system() == 'FreeBSD':
-			cmd = ["crontab", "-"]
-		s = subprocess.Popen(cmd, stdin=subprocess.PIPE)
-		s.stdin.write(current_crontab)
-		s.stdin.write(line + b'\n')
-		s.stdin.close()
-
-
-def read_crontab():
-	s = subprocess.Popen(["crontab", "-l"], stdin=subprocess.PIPE, stdout=subprocess.PIPE)
-	out = s.stdout.read()
-	s.stdout.close()
-	return out
 
 
 def setup_sudoers(user):
@@ -530,7 +545,6 @@ def restart_supervisor_processes(bench_path='.', web_workers=False):
 
 
 def restart_systemd_processes(bench_path='.', web_workers=False):
-	from .config.common_site_config import get_config
 	bench_name = get_bench_name(bench_path)
 	exec_cmd('sudo systemctl stop -- $(systemctl show -p Requires {bench_name}.target | cut -d= -f2)'.format(bench_name=bench_name))
 	exec_cmd('sudo systemctl start -- $(systemctl show -p Requires {bench_name}.target | cut -d= -f2)'.format(bench_name=bench_name))
@@ -819,7 +833,7 @@ def update_translations_p(args):
 
 
 def download_translations_p():
-	pool = multiprocessing.Pool(4)
+	pool = multiprocessing.Pool(multiprocessing.cpu_count())
 
 	langs = get_langs()
 	apps = ('frappe', 'erpnext')
@@ -1114,3 +1128,36 @@ def find_parent_bench(path):
 		# NOTE: the os.path.split assumes that given path is absolute
 		parent_dir = os.path.split(path)[0]
 		return find_parent_bench(parent_dir)
+
+
+def generate_command_cache(bench_path='.'):
+	"""Caches all available commands (even custom apps) via Frappe
+	Default caching behaviour: generated the first time any command (for a specific bench directory)
+	"""
+
+	python = get_env_cmd('python', bench_path=bench_path)
+	sites_path = os.path.join(bench_path, 'sites')
+
+	if os.path.exists(bench_cache_file):
+		os.remove(bench_cache_file)
+
+	try:
+		output = get_cmd_output("{0} -m frappe.utils.bench_helper get-frappe-commands".format(python), cwd=sites_path)
+		with open(bench_cache_file, 'w') as f:
+			json.dump(eval(output), f)
+		return json.loads(output)
+
+	except subprocess.CalledProcessError as e:
+		if hasattr(e, "stderr"):
+			print(e.stderr.decode('utf-8'))
+
+
+def clear_command_cache(bench_path='.'):
+	"""Clears commands cached
+	Default invalidation behaviour: destroyed on each run of `bench update`
+	"""
+
+	if os.path.exists(bench_cache_file):
+		os.remove(bench_cache_file)
+	else:
+		print("Bench command cache doesn't exist in this folder!")

playbooks/roles/wkhtmltopdf/tasks/main.yml

@@ -72,7 +72,7 @@
   get_url:
     url: https://github.com/wkhtmltopdf/wkhtmltopdf/releases/download/0.12.5/wkhtmltox_0.12.5-1.stretch_{{ "amd64" if ansible_architecture == "x86_64" else "i386"}}.deb
     dest: /tmp/wkhtmltox.deb
-    checksum: '{{ 1140b0ab02aa6e17346af2f14ed0de807376de475ba90e1db3975f112fbd20bb if ansible_architecture == "x86_64" else 5b2d15e738ac479e7a8ca6fd765f406c3684a48091813520f87878278d6dd22a }}'
+    checksum: "sha256:{{ '1140b0ab02aa6e17346af2f14ed0de807376de475ba90e1db3975f112fbd20bb' if ansible_architecture == 'x86_64' else '5b2d15e738ac479e7a8ca6fd765f406c3684a48091813520f87878278d6dd22a' }}"
   when: ansible_distribution == 'Debian' and ansible_distribution_major_version == '9'
 
 - name: download wkthmltox Debian 10