accounting/bench
2
0
Fork 0
mirror of https://github.com/frappe/bench.git synced 2024-11-11 15:51:03 +00:00
Code Issues Releases Activity

Merge branch 'master' into develop

Browse Source
This commit is contained in:
Rushabh Mehta 2017-05-16 11:38:06 +05:30 committed by GitHub
commit a309747f24
33 changed files with 587 additions and 337 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

162
README.md
View File

@ -1,43 +1,103 @@
Bench
=====
# Bench
The bench allows you to setup Frappe / ERPNext apps on your local Linux (CentOS 6, Debian 7, Ubuntu, etc) machine or a production server. You can use the bench to serve multiple frappe sites. If you are using a DigitalOcean droplet or any other VPS / Dedicated Server, make sure it has >= 1Gb of ram or has swap setup properly.
[![Build Status](https://travis-ci.org/frappe/bench.svg?branch=master)](https://travis-ci.org/frappe/bench)
The bench is a command-line utility that helps you to install apps, manage multiple sites and update Frappe / ERPNext apps on */nix (CentOS 6, Debian 7, Ubuntu, etc) for development and production. Bench will also create nginx and supervisor config files, setup backups and much more.
If you are using on a VPS make sure it has >= 1Gb of RAM or has swap setup properly.
To do this install, you must have basic information on how Linux works and should be able to use the command-line. If you are looking easier ways to get started and evaluate ERPNext, [download the Virtual Machine](https://erpnext.com/download) or take [a free trial on erpnext.com](https://erpnext.com/pricing).
If you have questions, please ask them on our [forum](https://discuss.erpnext.com/).
If you have questions, please ask them on the [forum](https://discuss.erpnext.com/).
Installation
============
## Installation
Easy Setup
----------
## Manual Install
- This is an opinionated setup with logging and SE Linux. So, it is best to setup on a blank server.
To manually install frappe/erpnext here are the steps
#### 1. Install Pre-requisites
- Python 2.7
- MariaDB 10+
- Nginx (for production)
- Nodejs
- Redis
- cron (crontab is required)
- wkhtmltopdf with patched Qt (for pdf generation)
#### 2. Install Bench
Install bench as a *non root* user,
git clone https://github.com/frappe/bench bench-repo
sudo pip install -e bench-repo
Note: Please do not remove the bench directory the above commands will create
#### Basic Usage
* Create a new bench
The init command will create a bench directory with frappe framework
installed. It will be setup for periodic backups and auto updates once
a day.
bench init frappe-bench && cd frappe-bench
* Add a site
Frappe apps are run by frappe sites and you will have to create at least one
site. The new-site command allows you to do that.
bench new-site site1.local
* Add apps
The get-app command gets remote frappe apps from a remote git repository and installs them. Example: [erpnext](https://github.com/frappe/erpnext)
bench get-app erpnext https://github.com/frappe/erpnext
* Install apps
To install an app on your new site, use the bench `install-app` command.
bench --site site1.local install-app erpnext
* Start bench
To start using the bench, use the `bench start` command
bench start
To login to Frappe / ERPNext, open your browser and go to `[your-external-ip]:8000`, probably `localhost:8000`
The default username is "Administrator" and password is what you set when you created the new site.
---
## Easy Install
- This is an opinionated setup so it is best to setup on a blank server.
- Works on Ubuntu 14.04 to 16.04, CentOS 7+, Debian 7 to 8 and MacOS X.
- You may have to install Python 2.7 (eg on Ubuntu 16.04+) by running `apt-get install python-minimal`
- This script will install the pre-requisites, install bench and setup an ERPNext site
- Passwords for Frappe Administrator and MariaDB (root) will be asked
- You can then login as **Administrator** with the Administrator password
- If you find any problems, post them on our forum: [https://discuss.erpnext.com](https://discuss.erpnext.com)
Production vs Develop
---------------------
*Production* setup should be run on a new box and installs nginx and supervisor to manage the processes. *Develop* setup uses `honcho` to manage the processes and uses the built-in web server (`bench start`)
Steps
-----
- If you find any problems, post them on the forum: [https://discuss.erpnext.com](https://discuss.erpnext.com)
Open your Terminal and enter:
#### Linux:
#### 1. Download the install script
For Linux:
wget https://raw.githubusercontent.com/frappe/bench/master/playbooks/install.py
#### MacOS:
For Mac OSX:
Install X Code (from App store) and HomeBrew (http://brew.sh/)
Install X Code (from App store) and HomeBrew (http://brew.sh/) first
brew install python
brew install git
@ -46,49 +106,40 @@ Download the Script
curl "https://raw.githubusercontent.com/frappe/bench/master/playbooks/install.py" -o install.py
#### Run the Script
#### 2. Run the install script
# For development
sudo python install.py --develop
If you are on a fresh server and logged in as root, use --user flag to create a user and install using that user
# For production
sudo python install.py --production
# If you're logged in as root, use --user flag to create a user and install using that user
python install.py --develop --user frappe
For developer setup:
For development, you have to explicitly start services by running `bench start`. This script requires Python2.7+ installed on your machine. You will have to manually create a new site (`bench new-site`) and get apps that you need (`bench get-app`, `bench install-app`).
sudo python install.py --develop
For production, you will have a preinstalled site with ERPNext installed in it.
For production:
You need to run this with a user that is **not** `root`, but can `sudo`. If you don't have such a user, you can search the web for *How to add a new user in { your OS }* and *How to add an existing user to sudoers in { your OS }*.
sudo python install.py --production
On Mac OS X, you will have to create a group with the same name as *{ your User }*. On creating this group, you have to assign *{ your User }* to it. You can do this by going to "System preferences" -> "Users & Groups" -> "+" (as if you were adding new account) -> Under "New account" select "Group" -> Type in group name -> "Create group"
#### What will this script do?
This script will:
- Install all the pre-requisites
- Install the command line `bench`
- Create a new bench (a folder that will contain your entire frappe/erpnext setup)
- Create a new site on the bench
- Install pre-requisites like git and ansible
- Shallow clones this bench repository under `/usr/local/frappe/bench-repo`
- Runs the Ansible playbook 'playbooks/develop/install.yml', which:
- Installs
- MariaDB and its config
- Redis
- NodeJS
- WKHTMLtoPDF with patched QT
- Initializes a new Bench at `~/frappe/frappe-bench` with `frappe` framework already installed under `apps`.
#### How do I start ERPNext
####Script Options:
```
--help
--verbose
--develop
--production
--site
--user
--bench-branch
--repo-url
```
1. For development: Go to your bench folder (`frappe-bench` by default) and start the bench with `bench start`
2. For production: Your process will be setup and managed by `nginx` and `supervisor`. [Setup Production](https://frappe.github.io/frappe/user/en/bench/guides/setup-production.html)
---
Help
====
For bench help, you can type
bench --help
Updating
========
@ -108,17 +159,16 @@ You can also run the parts of the bench selectively.
`bench update --requirements` will only update dependencies (python packages) for the apps installed
Guides
=======
- [Configuring HTTPS](https://frappe.github.io/frappe/user/en/bench/guides/configuring-https.html)
- [Using Let's Encrypt to setup HTTPS](https://frappe.github.io/frappe/user/en/bench/guides/lets-encrypt-ssl-setup.html)
- [Diagnosing the Scheduler](https://frappe.github.io/frappe/user/en/bench/guides/diagnosing-the-scheduler.html)
- [Change Hostname](https://frappe.github.io/frappe/user/en/bench/guides/how-to-change-host-name-from-localhost.html)
- [Change Hostname](https://frappe.github.io/frappe/user/en/bench/guides/adding-custom-domains)
- [Manual Setup](https://frappe.github.io/frappe/user/en/bench/guides/manual-setup.html)
- [Setup Production](https://frappe.github.io/frappe/user/en/bench/guides/setup-production.html)
- [Setup Multitenancy](https://frappe.github.io/frappe/user/en/bench/guides/setup-multitenancy.html)
- [Stopping Production](https://frappe.github.io/frappe/user/en/bench/guides/stop-production-and-start-development.html)
- [Stopping Production](https://github.com/frappe/bench/wiki/Stopping-Production-and-starting-Development)
Resources

60
bench/app.py
View File

@ -49,6 +49,9 @@ def write_appstxt(apps, bench_path='.'):
return f.write('\n'.join(apps))
def get_app(git_url, branch=None, bench_path='.', build_asset_files=True, verbose=False):
#less verbose app install
if '/' not in git_url:
git_url = 'https://github.com/frappe/' + git_url
#Gets repo name from URL
repo_name = git_url.rsplit('/', 1)[1].rsplit('.', 1)[0]
logger.info('getting app {}'.format(repo_name))
@ -69,7 +72,7 @@ def get_app(git_url, branch=None, bench_path='.', build_asset_files=True, verbos
apps_path = os.path.join(os.path.abspath(bench_path), 'apps')
os.rename(os.path.join(apps_path, repo_name), os.path.join(apps_path, app_name))
print 'installing', app_name
print('installing', app_name)
install_app(app=app_name, bench_path=bench_path, verbose=verbose)
if build_asset_files:
@ -92,20 +95,21 @@ def new_app(app, bench_path='.'):
run_frappe_cmd('make-app', apps, app, bench_path=bench_path)
install_app(app, bench_path=bench_path)
def install_app(app, bench_path='.', verbose=False):
def install_app(app, bench_path='.', verbose=False, no_cache=False):
logger.info('installing {}'.format(app))
# find_links = '--find-links={}'.format(conf.get('wheel_cache_dir')) if conf.get('wheel_cache_dir') else ''
find_links = ''
exec_cmd("{pip} install {quiet} {find_links} -e {app}".format(
exec_cmd("{pip} install {quiet} {find_links} -e {app} {no_cache}".format(
pip=os.path.join(bench_path, 'env', 'bin', 'pip'),
quiet="-q" if not verbose else "",
no_cache='--no-cache-dir' if not no_cache else '',
app=os.path.join(bench_path, 'apps', app),
find_links=find_links))
add_to_appstxt(app, bench_path=bench_path)
def remove_app(app, bench_path='.'):
if not app in get_apps(bench_path):
print "No app named {0}".format(app)
print("No app named {0}".format(app))
sys.exit(1)
app_path = os.path.join(bench_path, 'apps', app)
@ -117,7 +121,7 @@ def remove_app(app, bench_path='.'):
if os.path.exists(req_file):
out = subprocess.check_output(["bench", "--site", site, "list-apps"], cwd=bench_path)
if re.search(r'\b' + app + r'\b', out):
print "Cannot remove, app is installed on site: {0}".format(site)
print("Cannot remove, app is installed on site: {0}".format(site))
sys.exit(1)
exec_cmd(["{0} uninstall -y {1}".format(pip, app_path)])
@ -128,16 +132,42 @@ def remove_app(app, bench_path='.'):
restart_supervisor_processes(bench_path=bench_path)
def pull_all_apps(bench_path='.'):
def pull_all_apps(bench_path='.', reset=False):
'''Check all apps if there no local changes, pull'''
rebase = '--rebase' if get_config(bench_path).get('rebase_on_pull') else ''
# chech for local changes
if not reset:
for app in get_apps(bench_path=bench_path):
app_dir = get_repo_dir(app, bench_path=bench_path)
if os.path.exists(os.path.join(app_dir, '.git')):
out = subprocess.check_output(["git", "status"], cwd=app_dir)
if not re.search(r'nothing to commit, working (directory|tree) clean', out):
print('''
Cannot proceed with update: You have local changes in app "{0}" that are not committed.
Here are your choices:
1. Merge the {0} app manually with "git pull" / "git pull --rebase" and fix conflicts.
1. Temporarily remove your changes with "git stash" or discard them completely
with "bench update --reset" or for individual repositries "git reset --hard"
2. If your changes are helpful for others, send in a pull request via GitHub and
wait for them to be merged in the core.'''.format(app))
sys.exit(1)
for app in get_apps(bench_path=bench_path):
app_dir = get_repo_dir(app, bench_path=bench_path)
if os.path.exists(os.path.join(app_dir, '.git')):
remote = get_remote(app)
logger.info('pulling {0}'.format(app))
exec_cmd("git pull {rebase} {remote} {branch}".format(rebase=rebase,
remote=remote, branch=get_current_branch(app, bench_path=bench_path)), cwd=app_dir)
if reset:
exec_cmd("git fetch --all", cwd=app_dir)
exec_cmd("git reset --hard {remote}/{branch}".format(
remote=remote, branch=get_current_branch(app,bench_path=bench_path)), cwd=app_dir)
else:
exec_cmd("git pull {rebase} {remote} {branch}".format(rebase=rebase,
remote=remote, branch=get_current_branch(app, bench_path=bench_path)), cwd=app_dir)
exec_cmd('find . -name "*.pyc" -delete', cwd=app_dir)
@ -208,7 +238,7 @@ def get_upstream_version(app, branch=None, bench_path='.'):
branch = get_current_branch(app, bench_path=bench_path)
try:
contents = subprocess.check_output(['git', 'show', 'upstream/{branch}:{app}/__init__.py'.format(branch=branch, app=app)], cwd=repo_dir, stderr=subprocess.STDOUT)
except subprocess.CalledProcessError, e:
except subprocess.CalledProcessError as e:
if "Invalid object" in e.output:
return None
else:
@ -224,7 +254,7 @@ def get_repo_dir(app, bench_path='.'):
def switch_branch(branch, apps=None, bench_path='.', upgrade=False, check_upgrade=True):
from .utils import update_requirements, backup_all_sites, patch_sites, build_assets, pre_upgrade, post_upgrade
import utils
from . import utils
apps_dir = os.path.join(bench_path, 'apps')
version_upgrade = (False,)
switched_apps = []
@ -243,7 +273,7 @@ def switch_branch(branch, apps=None, bench_path='.', upgrade=False, check_upgrad
version_upgrade = is_version_upgrade(app=app, bench_path=bench_path, branch=branch)
if version_upgrade[0] and not upgrade:
raise MajorVersionUpgradeException("Switching to {0} will cause upgrade from {1} to {2}. Pass --upgrade to confirm".format(branch, version_upgrade[1], version_upgrade[2]), version_upgrade[1], version_upgrade[2])
print "Switching for "+app
print("Switching for "+app)
unshallow = "--unshallow" if os.path.exists(os.path.join(app_dir, ".git", "shallow")) else ""
exec_cmd("git config --unset-all remote.upstream.fetch", cwd=app_dir)
exec_cmd("git config --add remote.upstream.fetch '+refs/heads/*:refs/remotes/upstream/*'", cwd=app_dir)
@ -252,14 +282,14 @@ def switch_branch(branch, apps=None, bench_path='.', upgrade=False, check_upgrad
exec_cmd("git merge upstream/{branch}".format(branch=branch), cwd=app_dir)
switched_apps.append(app)
except CommandFailedError:
print "Error switching to branch {0} for {1}".format(branch, app)
print("Error switching to branch {0} for {1}".format(branch, app))
except InvalidRemoteException:
print "Remote does not exist for app "+app
print("Remote does not exist for app "+app)
except InvalidBranchException:
print "Branch {0} does not exist in Upstream for {1}".format(branch, app)
print("Branch {0} does not exist in Upstream for {1}".format(branch, app))
if switched_apps:
print "Successfully switched branches for:\n" + "\n".join(switched_apps)
print("Successfully switched branches for:\n" + "\n".join(switched_apps))
if version_upgrade[0] and upgrade:
update_requirements()

12
bench/cli.py Normal file → Executable file
View File

@ -26,9 +26,9 @@ def cli():
return frappe_cmd()
elif len(sys.argv) > 1 and sys.argv[1]=="--help":
print click.Context(bench_command).get_help()
print
print get_frappe_help()
print(click.Context(bench_command).get_help())
print()
print(get_frappe_help())
return
elif len(sys.argv) > 1 and sys.argv[1] in get_apps():
@ -43,11 +43,11 @@ def cli():
def check_uid():
if cmd_requires_root() and not is_root():
print 'superuser privileges required for this command'
print('superuser privileges required for this command')
sys.exit(1)
def cmd_requires_root():
if len(sys.argv) > 2 and sys.argv[2] in ('production', 'sudoers', 'lets-encrypt', 'fonts', 'reload-nginx'):
if len(sys.argv) > 2 and sys.argv[2] in ('production', 'sudoers', 'lets-encrypt', 'fonts', 'reload-nginx', 'firewall', 'ssh-port'):
return True
if len(sys.argv) >= 2 and sys.argv[1] in ('patch', 'renew-lets-encrypt', 'disable-production'):
return True
@ -69,7 +69,7 @@ def change_uid():
drop_privileges(uid_name=frappe_user, gid_name=frappe_user)
os.environ['HOME'] = pwd.getpwnam(frappe_user).pw_dir
else:
print 'You should not run this command as root'
print('You should not run this command as root')
sys.exit(1)
def old_frappe_cli(bench_path='.'):

2
bench/commands/config.py
View File

@ -64,7 +64,7 @@ def config_http_timeout(seconds):
@click.command('set-common-config')
@click.option('configs', '-c', '--config', multiple=True, type=(unicode, unicode))
@click.option('configs', '-c', '--config', multiple=True, type=(str, str))
def set_common_config(configs):
import ast
from bench.config.common_site_config import update_config

2
bench/commands/git.py
View File

@ -29,5 +29,5 @@ def remote_urls():
if os.path.exists(os.path.join(repo_dir, '.git')):
remote = get_remote(app)
remote_url = subprocess.check_output(['git', 'config', '--get', 'remote.{}.url'.format(remote)], cwd=repo_dir).strip()
print "{app} {remote_url}".format(app=app, remote_url=remote_url)
print("{app} {remote_url}".format(app=app, remote_url=remote_url))

34
bench/commands/setup.py Normal file → Executable file
View File

@ -75,13 +75,33 @@ def setup_env():
from bench.utils import setup_env
setup_env()
@click.command('firewall')
def setup_firewall():
"Setup firewall"
from bench.utils import run_playbook
click.confirm('Setting up the firewall will block all ports except 80, 443 and 22\n'
'Do you want to continue?',
abort=True)
run_playbook('production/setup_firewall.yml')
@click.command('ssh-port')
@click.argument('port')
def set_ssh_port(port):
"Set SSH Port"
from bench.utils import run_playbook
click.confirm('This will change your SSH Port to {}\n'
'Do you want to continue?'.format(port),
abort=True)
run_playbook('production/change_ssh_port.yml', {"ssh_port": port})
@click.command('lets-encrypt')
@click.argument('site')
def setup_letsencrypt(site):
@click.option('--custom-domain')
def setup_letsencrypt(site, custom_domain):
"Setup lets-encrypt for site"
from bench.config.lets_encrypt import setup_letsencrypt
setup_letsencrypt(site, bench_path='.')
setup_letsencrypt(site, custom_domain, bench_path='.')
@click.command('procfile')
@ -115,7 +135,7 @@ def add_domain(domain, site=None, ssl_certificate=None, ssl_certificate_key=None
from bench.config.site_config import add_domain
if not site:
print "Please specify site"
print("Please specify site")
sys.exit(1)
add_domain(site, domain, ssl_certificate, ssl_certificate_key, bench_path='.')
@ -128,7 +148,7 @@ def remove_domain(domain, site=None):
from bench.config.site_config import remove_domain
if not site:
print "Please specify site"
print("Please specify site")
sys.exit(1)
remove_domain(site, domain, bench_path='.')
@ -140,12 +160,12 @@ def sync_domains(domains, site=None):
from bench.config.site_config import sync_domains
if not site:
print "Please specify site"
print("Please specify site")
sys.exit(1)
domains = json.loads(domains)
if not isinstance(domains, list):
print "Domains should be a json list of strings or dictionaries"
print("Domains should be a json list of strings or dictionaries")
sys.exit(1)
changed = sync_domains(site, domains, bench_path='.')
@ -170,3 +190,5 @@ setup.add_command(setup_fonts)
setup.add_command(add_domain)
setup.add_command(remove_domain)
setup.add_command(sync_domains)
setup.add_command(setup_firewall)
setup.add_command(set_ssh_port)

60
bench/commands/update.py
View File

@ -15,10 +15,11 @@ from bench import patches
@click.option('--requirements',is_flag=True, help="Update requirements")
@click.option('--restart-supervisor',is_flag=True, help="restart supervisor processes after update")
@click.option('--auto',is_flag=True)
@click.option('--upgrade',is_flag=True)
@click.option('--upgrade',is_flag=True, help="Required for major version updates")
@click.option('--no-backup',is_flag=True)
@click.option('--force',is_flag=True)
def update(pull=False, patch=False, build=False, bench=False, auto=False, restart_supervisor=False, requirements=False, no_backup=False, upgrade=False, force=False):
@click.option('--reset', is_flag=True, help="Hard resets git branch's to their new states overriding any changes and overriding rebase on pull")
def update(pull=False, patch=False, build=False, bench=False, auto=False, restart_supervisor=False, requirements=False, no_backup=False, upgrade=False, force=False, reset=False):
"Update bench"
if not (pull or patch or build or bench or requirements):
@ -39,28 +40,29 @@ def update(pull=False, patch=False, build=False, bench=False, auto=False, restar
'requirements': requirements,
'no-backup': no_backup,
'restart-supervisor': restart_supervisor,
'upgrade': upgrade
'upgrade': upgrade,
'reset':reset
})
if conf.get('release_bench'):
print 'Release bench, cannot update'
print('Release bench, cannot update')
sys.exit(1)
version_upgrade = is_version_upgrade()
if version_upgrade[0] and not upgrade:
print
print
print "This update will cause a major version change in Frappe/ERPNext from {0} to {1}.".format(*version_upgrade[1:])
print "This would take significant time to migrate and might break custom apps. Please run `bench update --upgrade` to confirm."
print
print "You can stay on the latest stable release by running `bench switch-to-master` or pin your bench to {0}".format(version_upgrade[1])
print()
print()
print("This update will cause a major version change in Frappe/ERPNext from {0} to {1}.".format(*version_upgrade[1:]))
print("This would take significant time to migrate and might break custom apps. Please run `bench update --upgrade` to confirm.")
print()
print("You can stay on the latest stable release by running `bench switch-to-master` or pin your bench to {0} by running `bench switch-to-v{0}`".format(version_upgrade[1]))
sys.exit(1)
_update(pull, patch, build, bench, auto, restart_supervisor, requirements, no_backup, upgrade, force=force)
_update(pull, patch, build, bench, auto, restart_supervisor, requirements, no_backup, upgrade, force=force, reset=reset)
def _update(pull=False, patch=False, build=False, update_bench=False, auto=False, restart_supervisor=False, requirements=False, no_backup=False, upgrade=False, bench_path='.', force=False):
def _update(pull=False, patch=False, build=False, update_bench=False, auto=False, restart_supervisor=False, requirements=False, no_backup=False, upgrade=False, bench_path='.', force=False, reset=False):
conf = get_config(bench_path=bench_path)
version_upgrade = is_version_upgrade(bench_path=bench_path)
@ -73,20 +75,25 @@ def _update(pull=False, patch=False, build=False, update_bench=False, auto=False
before_update(bench_path=bench_path, requirements=requirements)
if pull:
pull_all_apps(bench_path=bench_path)
pull_all_apps(bench_path=bench_path, reset=reset)
if requirements:
print('Updating Python libraries...')
update_requirements(bench_path=bench_path)
if upgrade and (version_upgrade[0] or (not version_upgrade[0] and force)):
pre_upgrade(version_upgrade[1], version_upgrade[2], bench_path=bench_path)
import bench.utils, bench.app
print('Reloading bench...')
reload(bench.utils)
reload(bench.app)
if patch:
if not no_backup:
print('Backing up sites...')
backup_all_sites(bench_path=bench_path)
print('Patching sites...')
patch_sites(bench_path=bench_path)
if build:
build_assets(bench_path=bench_path)
@ -95,9 +102,10 @@ def _update(pull=False, patch=False, build=False, update_bench=False, auto=False
if restart_supervisor or conf.get('restart_supervisor_on_update'):
restart_supervisor_processes(bench_path=bench_path)
print "_"*80
print "Bench: Open source installer + admin for Frappe and ERPNext (https://erpnext.com)"
print
print("_"*80)
print("Bench: Deployment tool for Frappe and ERPNext (https://erpnext.org).")
print("Open source depends on your contributions, so please contribute bug reports, patches, fixes or cash and be a part of the community")
print()
@click.command('retry-upgrade')
@ -110,7 +118,7 @@ def retry_upgrade(version):
def restart_update(kwargs):
args = ['--'+k for k, v in kwargs.items() if v]
args = ['--'+k for k, v in list(kwargs.items()) if v]
os.execv(sys.argv[0], sys.argv[:2] + args)
@ -122,8 +130,8 @@ def switch_to_branch(branch, apps, upgrade=False):
"Switch all apps to specified branch, or specify apps separated by space"
from bench.app import switch_to_branch
switch_to_branch(branch=branch, apps=list(apps), upgrade=upgrade)
print 'Switched to ' + branch
print 'Please run `bench update --patch` to be safe from any differences in database schema'
print('Switched to ' + branch)
print('Please run `bench update --patch` to be safe from any differences in database schema')
@click.command('switch-to-master')
@ -132,9 +140,9 @@ def switch_to_master(upgrade=False):
"Switch frappe and erpnext to master branch"
from bench.app import switch_to_master
switch_to_master(upgrade=upgrade, apps=['frappe', 'erpnext'])
print
print 'Switched to master'
print 'Please run `bench update --patch` to be safe from any differences in database schema'
print()
print('Switched to master')
print('Please run `bench update --patch` to be safe from any differences in database schema')
@click.command('switch-to-develop')
@ -143,6 +151,8 @@ def switch_to_develop(upgrade=False):
"Switch frappe and erpnext to develop branch"
from bench.app import switch_to_develop
switch_to_develop(upgrade=upgrade, apps=['frappe', 'erpnext'])
print
print 'Switched to develop'
print 'Please run `bench update --patch` to be safe from any differences in database schema'
print()
print('Switched to develop')
print('Please run `bench update --patch` to be safe from any differences in database schema')

14
bench/commands/utils.py
View File

@ -12,11 +12,11 @@ def start(no_dev, concurrency):
@click.command('restart')
@click.option('--web-workers', is_flag=True, default=False)
def restart(web_workers):
@click.option('--web', is_flag=True, default=False)
def restart(web):
"Restart supervisor processes"
from bench.utils import restart_supervisor_processes
restart_supervisor_processes(bench_path='.', web_workers=web_workers)
restart_supervisor_processes(bench_path='.', web_workers=web)
@click.command('set-nginx-port')
@ -86,10 +86,10 @@ def renew_lets_encrypt():
@click.command()
def shell(bench_path='.'):
if not os.environ.get('SHELL'):
print "Cannot get shell"
print("Cannot get shell")
sys.exit(1)
if not os.path.exists('sites'):
print "sites dir doesn't exist"
print("sites dir doesn't exist")
sys.exit(1)
env = copy.copy(os.environ)
env['PS1'] = '(' + os.path.basename(os.path.dirname(os.path.abspath(__file__))) + ')' + env.get('PS1', '')
@ -104,7 +104,7 @@ def backup_site(site):
"backup site"
from bench.utils import get_sites, backup_site
if not site in get_sites(bench_path='.'):
print 'site not found'
print('site not found')
sys.exit(1)
backup_site(site, bench_path='.')
@ -142,4 +142,4 @@ def disable_production():
def bench_src():
"""Prints bench source folder path, which can be used as: cd `bench src` """
import bench
print os.path.dirname(bench.__path__[0])
print(os.path.dirname(bench.__path__[0]))

13
bench/config/common_site_config.py
View File

@ -1,4 +1,9 @@
import os, multiprocessing, getpass, json, urlparse
import os, multiprocessing, getpass, json
try:
from urllib.parse import urlparse
except ImportError:
from urlparse import urlparse
default_config = {
'restart_supervisor_on_update': False,
@ -81,19 +86,19 @@ def make_ports(bench_path):
bench_path = os.path.join(benches_path, folder)
if os.path.isdir(bench_path):
bench_config = get_config(bench_path)
for key in default_ports.keys():
for key in list(default_ports.keys()):
value = bench_config.get(key)
# extract port from redis url
if value and (key in ('redis_cache', 'redis_queue', 'redis_socketio')):
value = urlparse.urlparse(value).port
value = urlparse(value).port
if value:
existing_ports.setdefault(key, []).append(value)
# new port value = max of existing port value + 1
ports = {}
for key, value in default_ports.items():
for key, value in list(default_ports.items()):
existing_value = existing_ports.get(key, [])
if existing_value:
value = max(existing_value) + 1

62
bench/config/lets_encrypt.py
View File

@ -1,65 +1,87 @@
import bench, os, click, errno, urllib
import bench, os, click, errno
from bench.utils import exec_cmd, CommandFailedError
from bench.config.site_config import update_site_config
from bench.config.site_config import update_site_config, remove_domain, get_domains
from bench.config.nginx import make_nginx_conf
from bench.config.production_setup import service
from bench.config.common_site_config import get_config
from crontab import CronTab
def setup_letsencrypt(site, bench_path):
try:
from urllib.request import urlretrieve
except ImportError:
from urllib import urlretrieve
def setup_letsencrypt(site, custom_domain, bench_path):
site_path = os.path.join(bench_path, "sites", site, "site_config.json")
if not os.path.exists(os.path.dirname(site_path)):
print "No site named "+site
print("No site named "+site)
return
if custom_domain:
domains = get_domains(site, bench_path)
for d in domains:
if (isinstance(d, dict) and d['domain']==custom_domain):
print("SSL for Domain {0} already exists".format(custom_domain))
return
if not custom_domain in domains:
print("No custom domain named {0} set for site".format(custom_domain))
return
click.confirm('Running this will stop the nginx service temporarily causing your sites to go offline\n'
'Do you want to continue?',
abort=True)
if not get_config(bench_path).get("dns_multitenant"):
print "You cannot setup SSL without DNS Multitenancy"
print("You cannot setup SSL without DNS Multitenancy")
return
create_config(site)
run_certbot_and_setup_ssl(site, bench_path)
create_config(site, custom_domain)
run_certbot_and_setup_ssl(site, custom_domain, bench_path)
setup_crontab()
def create_config(site):
config = bench.env.get_template('letsencrypt.cfg').render(domain=site)
config_path = '/etc/letsencrypt/configs/{site}.cfg'.format(site=site)
def create_config(site, custom_domain):
config = bench.env.get_template('letsencrypt.cfg').render(domain=custom_domain or site)
config_path = '/etc/letsencrypt/configs/{site}.cfg'.format(site=custom_domain or site)
create_dir_if_missing(config_path)
with open(config_path, 'w') as f:
f.write(config)
def run_certbot_and_setup_ssl(site, bench_path):
def run_certbot_and_setup_ssl(site, custom_domain, bench_path):
service('nginx', 'stop')
get_certbot()
try:
exec_cmd("{path} --config /etc/letsencrypt/configs/{site}.cfg certonly".format(path=get_certbot_path(), site=site))
exec_cmd("{path} --config /etc/letsencrypt/configs/{site}.cfg certonly".format(path=get_certbot_path(), site=custom_domain or site))
except CommandFailedError:
service('nginx', 'start')
print "There was a problem trying to setup SSL for your site"
print("There was a problem trying to setup SSL for your site")
return
ssl_path = "/etc/letsencrypt/live/{site}/".format(site=site)
ssl_path = "/etc/letsencrypt/live/{site}/".format(site=custom_domain or site)
ssl_config = { "ssl_certificate": os.path.join(ssl_path, "fullchain.pem"),
"ssl_certificate_key": os.path.join(ssl_path, "privkey.pem") }
update_site_config(site, ssl_config, bench_path=bench_path)
make_nginx_conf(bench_path)
if custom_domain:
remove_domain(site, custom_domain, bench_path)
domains = get_domains(site, bench_path)
ssl_config['domain'] = custom_domain
domains.append(ssl_config)
update_site_config(site, { "domains": domains }, bench_path=bench_path)
else:
update_site_config(site, ssl_config, bench_path=bench_path)
make_nginx_conf(bench_path)
service('nginx', 'start')
def setup_crontab():
job_command = 'sudo service nginx stop && /opt/certbot-auto renew && sudo service nginx start'
user_crontab = CronTab(user=True)
user_crontab = CronTab()
if job_command not in str(user_crontab):
job = user_crontab.new(command=job_command, comment="Renew lets-encrypt every month")
job.every().month()
@ -77,8 +99,8 @@ def get_certbot():
create_dir_if_missing(certbot_path)
if not os.path.isfile(certbot_path):
urllib.urlretrieve ("https://dl.eff.org/certbot-auto", certbot_path)
os.chmod(certbot_path, 0744)
urlretrieve ("https://dl.eff.org/certbot-auto", certbot_path)
os.chmod(certbot_path, 0o744)
def get_certbot_path():

4
bench/config/nginx.py
View File

@ -25,7 +25,7 @@ def make_nginx_conf(bench_path, yes=False):
"error_pages": get_error_pages(),
"allow_rate_limiting": allow_rate_limiting,
# for nginx map variable
"random_string": "".join(random.choice(string.ascii_lowercase) for i in xrange(7))
"random_string": "".join(random.choice(string.ascii_lowercase) for i in range(7))
}
if allow_rate_limiting:
@ -159,7 +159,7 @@ def get_sites_with_config(bench_path):
if dns_multitenant and site_config.get('domains'):
for domain in site_config.get('domains'):
# domain can be a string or a dict with 'domain', 'ssl_certificate', 'ssl_certificate_key'
if isinstance(domain, basestring):
if isinstance(domain, str) or isinstance(domain, unicode):
domain = { 'domain': domain }
domain['name'] = site

2
bench/config/production_setup.py
View File

@ -66,7 +66,7 @@ def service(service, option):
exec_cmd(service_manager_command)
else:
raise Exception, 'No service manager found'
raise Exception('No service manager found')
def get_supervisor_confdir():
possiblities = ('/etc/supervisor/conf.d', '/etc/supervisor.d/', '/etc/supervisord/conf.d', '/etc/supervisord.d')

22
bench/config/redis.py
View File

@ -1,13 +1,18 @@
from .common_site_config import get_config
import re, os, subprocess, urlparse, semantic_version
import re, os, subprocess, semantic_version
import bench
try:
from urllib.parse import urlparse
except ImportError:
from urlparse import urlparse
def generate_config(bench_path):
config = get_config(bench_path)
ports = {}
for key in ('redis_cache', 'redis_queue', 'redis_socketio'):
ports[key] = urlparse.urlparse(config[key]).port
ports[key] = urlparse(config[key]).port
write_redis_config(
template_name='redis_queue.conf',
@ -29,7 +34,7 @@ def generate_config(bench_path):
write_redis_config(
template_name='redis_cache.conf',
context={
"maxmemory": config.get('cache_maxmemory', '50'),
"maxmemory": config.get('cache_maxmemory', get_max_redis_memory()),
"port": ports['redis_cache'],
"redis_version": get_redis_version(),
},
@ -60,3 +65,14 @@ def get_redis_version():
version = semantic_version.Version(version[0], partial=True)
return float('{major}.{minor}'.format(major=version.major, minor=version.minor))
def get_max_redis_memory():
import psutil
total_virtual_mem = psutil.virtual_memory().total/(pow(1024, 2))
max_memory = int(total_virtual_mem * 0.05) # Max memory for redis is 5% of virtual memory
if max_memory < 50:
return 50
else:
return max_memory

6
bench/config/site_config.py
View File

@ -43,7 +43,7 @@ def add_domain(site, domain, ssl_certificate, ssl_certificate_key, bench_path='.
domains = get_domains(site, bench_path)
for d in domains:
if (isinstance(d, dict) and d['domain']==domain) or d==domain:
print "Domain {0} already exists".format(domain)
print("Domain {0} already exists".format(domain))
return
if ssl_certificate_key and ssl_certificate:
@ -75,7 +75,7 @@ def sync_domains(site, domains, bench_path='.'):
changed = True
else:
for d in existing_domains.values():
for d in list(existing_domains.values()):
if d != new_domains.get(d['domain']):
changed = True
break
@ -92,7 +92,7 @@ def get_domains(site, bench_path='.'):
def get_domains_dict(domains):
domains_dict = defaultdict(dict)
for d in domains:
if isinstance(d, basestring):
if isinstance(d, str):
domains_dict[d] = { 'domain': d }
elif isinstance(d, dict):

2
bench/patches/patches.txt
View File

@ -2,3 +2,5 @@ bench.patches.v3.deprecate_old_config
bench.patches.v3.celery_to_rq
bench.patches.v3.redis_bind_ip
bench.patches.v4.update_node
bench.patches.v4.update_socketio

15
bench/patches/v4/update_node.py
View File

@ -12,9 +12,10 @@ def execute(bench_path):
else:
click.echo('''
No node executable was found on your machine.
Please install node 5.x before running "bench update".
Installation instructions for CentOS and Ubuntu can be found on the following link,
"https://www.metachris.com/2015/10/how-to-install-nodejs-5-on-centos-and-ubuntu/"
Please install latest node version before running "bench update". For installation instructions
please refer "Debian and Ubuntu based Linux distributions" section or "Enterprise Linux and
Fedora" section depending upon your OS on the following link,
"https://nodejs.org/en/download/package-manager/"
''')
sys.exit(1)
@ -22,8 +23,10 @@ def execute(bench_path):
if node_ver < expected_node_ver:
click.echo('''
Please update node version to 5.x before running "bench update".
Installation instructions for CentOS and Ubuntu can be found on the following link,
"https://www.metachris.com/2015/10/how-to-install-nodejs-5-on-centos-and-ubuntu/"
Please update node to latest version before running "bench update".
Please install latest node version before running "bench update". For installation instructions
please refer "Debian and Ubuntu based Linux distributions" section or "Enterprise Linux and
Fedora" section depending upon your OS on the following link,
"https://nodejs.org/en/download/package-manager/"
''')
sys.exit(1)

4
bench/patches/v4/update_socketio.py Normal file
View File

@ -0,0 +1,4 @@
import subprocess
def execute(bench_path):
subprocess.check_output(['npm', 'install', 'socket.io'])

60
bench/release.py
View File

@ -29,7 +29,7 @@ def release(bench_path, app, bump_type, develop='develop', master='master',
def validate(bench_path):
config = get_config(bench_path)
if not config.get('release_bench'):
print 'bench not configured to release'
print('bench not configured to release')
sys.exit(1)
global github_username, github_password
@ -38,7 +38,7 @@ def validate(bench_path):
github_password = config.get('github_password')
if not github_username:
github_username = raw_input('Username: ')
github_username = input('Username: ')
if not github_password:
github_password = getpass.getpass()
@ -54,12 +54,12 @@ def bump(bench_path, app, bump_type, develop, master, remote, owner, repo_name=N
message = get_release_message(repo_path, develop=develop, master=master, remote=remote)
if not message:
print 'No commits to release'
print('No commits to release')
return
print
print message
print
print()
print(message)
print()
click.confirm('Do you want to continue?', abort=True)
@ -68,7 +68,7 @@ def bump(bench_path, app, bump_type, develop, master, remote, owner, repo_name=N
tag_name = create_release(repo_path, new_version, develop=develop, master=master)
push_release(repo_path, develop=develop, master=master, remote=remote)
create_github_release(repo_path, tag_name, message, remote=remote, owner=owner, repo_name=repo_name)
print 'Released {tag} for {repo_path}'.format(tag=tag_name, repo_path=repo_path)
print('Released {tag} for {repo_path}'.format(tag=tag_name, repo_path=repo_path))
def update_branches_and_check_for_changelog(repo_path, develop='develop', master='master', remote='upstream'):
@ -81,7 +81,7 @@ def update_branches_and_check_for_changelog(repo_path, develop='develop', master
check_for_unmerged_changelog(repo_path)
def update_branch(repo_path, branch, remote):
print "updating local branch of", repo_path, 'using', remote + '/' + branch
print("updating local branch of", repo_path, 'using', remote + '/' + branch)
repo = git.Repo(repo_path)
g = repo.git
@ -95,7 +95,7 @@ def check_for_unmerged_changelog(repo_path):
raise Exception("Unmerged change log! in " + repo_path)
def get_release_message(repo_path, develop='develop', master='master', remote='upstream'):
print 'getting release message for', repo_path, 'comparing', master, '...', develop
print('getting release message for', repo_path, 'comparing', master, '...', develop)
repo = git.Repo(repo_path)
g = repo.git
@ -109,7 +109,7 @@ def bump_repo(repo_path, bump_type, develop='develop', master='master'):
current_version = get_current_version(repo_path)
new_version = get_bumped_version(current_version, bump_type)
print 'bumping version from', current_version, 'to', new_version
print('bumping version from', current_version, 'to', new_version)
set_version(repo_path, new_version)
return new_version
@ -154,7 +154,7 @@ def get_bumped_version(version, bump_type):
else:
v.prerelease[1] = str(int(v.prerelease[1]) + 1)
return unicode(v)
return str(v)
def set_version(repo_path, version):
set_filename_version(os.path.join(repo_path, os.path.basename(repo_path),'__init__.py'), version, '__version__')
@ -192,7 +192,7 @@ def set_filename_version(filename, version_number, pattern):
f.write(contents)
def commit_changes(repo_path, new_version):
print 'committing version change to', repo_path
print('committing version change to', repo_path)
repo = git.Repo(repo_path)
app_name = os.path.basename(repo_path)
@ -200,13 +200,13 @@ def commit_changes(repo_path, new_version):
repo.index.commit('bumped to version {}'.format(new_version))
def create_release(repo_path, new_version, develop='develop', master='master'):
print 'creating release for version', new_version
print('creating release for version', new_version)
repo = git.Repo(repo_path)
g = repo.git
g.checkout(master)
try:
g.merge(develop, '--no-ff')
except git.exc.GitCommandError, e:
except git.exc.GitCommandError as e:
handle_merge_error(e, source=develop, target=master)
tag_name = 'v' + new_version
@ -215,29 +215,29 @@ def create_release(repo_path, new_version, develop='develop', master='master'):
try:
g.merge(master)
except git.exc.GitCommandError, e:
except git.exc.GitCommandError as e:
handle_merge_error(e, source=master, target=develop)
if develop != 'develop':
print 'merging master into develop'
print('merging master into develop')
g.checkout('develop')
try:
g.merge(master)
except git.exc.GitCommandError, e:
except git.exc.GitCommandError as e:
handle_merge_error(e, source=master, target='develop')
return tag_name
def handle_merge_error(e, source, target):
print '-'*80
print 'Error when merging {source} into {target}'.format(source=source, target=target)
print e
print 'You can open a new terminal, try to manually resolve the conflict/error and continue'
print '-'*80
print('-'*80)
print('Error when merging {source} into {target}'.format(source=source, target=target))
print(e)
print('You can open a new terminal, try to manually resolve the conflict/error and continue')
print('-'*80)
click.confirm('Have you manually resolved the error?', abort=True)
def push_release(repo_path, develop='develop', master='master', remote='upstream'):
print 'pushing branches', master, develop, 'of', repo_path
print('pushing branches', master, develop, 'of', repo_path)
repo = git.Repo(repo_path)
g = repo.git
args = [
@ -246,22 +246,22 @@ def push_release(repo_path, develop='develop', master='master', remote='upstream
]
if develop != 'develop':
print 'pushing develop branch of', repo_path
print('pushing develop branch of', repo_path)
args.append('develop:develop')
args.append('--tags')
print g.push(remote, *args)
print(g.push(remote, *args))
def create_github_release(repo_path, tag_name, message, remote='upstream', owner='frappe', repo_name=None,
gh_username=None, gh_password=None):
print 'creating release on github'
print('creating release on github')
global github_username, github_password
if not (gh_username and gh_password):
if not (github_username and github_password):
raise Exception, "No credentials"
raise Exception("No credentials")
gh_username = github_username
gh_password = github_password
@ -274,7 +274,7 @@ def create_github_release(repo_path, tag_name, message, remote='upstream', owner
'draft': False,
'prerelease': False
}
for i in xrange(3):
for i in range(3):
try:
r = requests.post('https://api.github.com/repos/{owner}/{repo_name}/releases'.format(
owner=owner, repo_name=repo_name),
@ -282,12 +282,12 @@ def create_github_release(repo_path, tag_name, message, remote='upstream', owner
r.raise_for_status()
break
except requests.exceptions.HTTPError:
print 'request failed, retrying....'
print('request failed, retrying....')
sleep(3*i + 1)
if i !=2:
continue
else:
print r.json()
print(r.json())
raise
return r

8
bench/tests/test_init.py
View File

@ -1,4 +1,4 @@
from __future__ import unicode_literals
import unittest
import json, os, shutil, subprocess
import bench
@ -178,7 +178,7 @@ class TestBenchInit(unittest.TestCase):
try:
subprocess.check_output(drop_site_cmd, cwd=bench_path)
except subprocess.CalledProcessError as err:
print err.output
print(err.output)
if not archived_sites_path:
archived_sites_path = os.path.join(bench_path, 'archived_sites')
@ -234,8 +234,8 @@ class TestBenchInit(unittest.TestCase):
config = self.load_json(common_site_config_path)
for key, value in expected_config.items():
self.assertEquals(config.get(key), value)
for key, value in list(expected_config.items()):
self.assertEqual(config.get(key), value)
def assert_exists(self, *args):
self.assertTrue(os.path.exists(os.path.join(*args)))

23
bench/tests/test_setup_production.py
View File

@ -1,4 +1,4 @@
from __future__ import unicode_literals
from bench.tests import test_init
from bench.config.production_setup import setup_production, get_supervisor_confdir, disable_production
import bench.utils
@ -36,23 +36,6 @@ class TestSetupProduction(test_init.TestBenchInit):
bench_path = os.path.join(os.path.abspath(self.benches_path), bench_name)
disable_production(bench_path)
def test_setup_production_v6(self):
bench_name = 'test-bench-v6'
self.test_init(bench_name, frappe_branch='v6.x.x')
user = getpass.getuser()
bench_path = os.path.join(os.path.abspath(self.benches_path), bench_name)
setup_production(user, bench_path)
self.assert_nginx_config(bench_name)
self.assert_nginx_process()
self.assert_supervisor_config(bench_name, use_rq=False)
self.assert_supervisor_process(bench_name, use_rq=False)
disable_production(bench_path)
def test_disable_production(self):
bench_name = 'test-disable-prod'
self.test_init(bench_name, frappe_branch='master')
@ -76,7 +59,7 @@ class TestSetupProduction(test_init.TestBenchInit):
self.assertTrue(os.path.exists(conf_dest))
# symlink matches
self.assertEquals(os.path.realpath(conf_dest), conf_src)
self.assertEqual(os.path.realpath(conf_dest), conf_src)
# file content
with open(conf_src, "r") as f:
@ -113,7 +96,7 @@ class TestSetupProduction(test_init.TestBenchInit):
self.assertTrue(os.path.exists(conf_dest))
# symlink matches
self.assertEquals(os.path.realpath(conf_dest), conf_src)
self.assertEqual(os.path.realpath(conf_dest), conf_src)
# file content
with open(conf_src, "r") as f:

85
bench/utils.py
View File

@ -16,8 +16,8 @@ folders_in_bench = ('apps', 'sites', 'config', 'logs', 'config/pids')
def get_frappe(bench_path='.'):
frappe = get_env_cmd('frappe', bench_path=bench_path)
if not os.path.exists(frappe):
print 'frappe app is not installed. Run the following command to install frappe'
print 'bench get-app https://github.com/frappe/frappe.git'
print('frappe app is not installed. Run the following command to install frappe')
print('bench get-app https://github.com/frappe/frappe.git')
return frappe
def get_env_cmd(cmd, bench_path='.'):
@ -33,7 +33,7 @@ def init(path, apps_path=None, no_procfile=False, no_backups=False,
from bench.patches import set_all_patches_executed
if os.path.exists(path):
print 'Directory {} already exists!'.format(path)
print('Directory {} already exists!'.format(path))
raise Exception("Site directory already exists")
# sys.exit(1)
@ -76,17 +76,17 @@ def init(path, apps_path=None, no_procfile=False, no_backups=False,
def clone_apps_from(bench_path, clone_from):
from .app import install_app
print 'Copying apps from {0}...'.format(clone_from)
print('Copying apps from {0}...'.format(clone_from))
subprocess.check_output(['cp', '-R', os.path.join(clone_from, 'apps'), bench_path])
print 'Copying node_modules from {0}...'.format(clone_from)
print('Copying node_modules from {0}...'.format(clone_from))
subprocess.check_output(['cp', '-R', os.path.join(clone_from, 'node_modules'), bench_path])
def setup_app(app):
# run git reset --hard in each branch, pull latest updates and install_app
app_path = os.path.join(bench_path, 'apps', app)
if os.path.exists(os.path.join(app_path, '.git')):
print 'Cleaning up {0}'.format(app)
print('Cleaning up {0}'.format(app))
# remove .egg-ino
subprocess.check_output(['rm', '-rf', app + '.egg-info'], cwd=app_path)
@ -116,6 +116,9 @@ def exec_cmd(cmd, cwd='.'):
stderr = stdout = subprocess.PIPE
else:
stderr = stdout = None
logger.info(cmd)
p = subprocess.Popen(cmd, cwd=cwd, shell=True, stdout=stdout, stderr=stderr)
if async:
@ -137,7 +140,6 @@ def setup_socketio(bench_path='.'):
exec_cmd("npm install socket.io redis express superagent cookie", cwd=bench_path)
def new_site(site, mariadb_root_password=None, admin_password=None, bench_path='.'):
import hashlib
logger.info('creating new site {}'.format(site))
mariadb_root_password_fragment = '--root_password {}'.format(mariadb_root_password) if mariadb_root_password else ''
admin_password_fragment = '--admin_password {}'.format(admin_password) if admin_password else ''
@ -238,7 +240,7 @@ def setup_sudoers(user):
f.write('\n#includedir /etc/sudoers.d\n')
if set_permissions:
os.chmod('/etc/sudoers', 0440)
os.chmod('/etc/sudoers', 0o440)
sudoers_file = '/etc/sudoers.d/frappe'
@ -255,7 +257,7 @@ def setup_sudoers(user):
with open(sudoers_file, 'w') as f:
f.write(frappe_sudoers.encode('utf-8'))
os.chmod(sudoers_file, 0440)
os.chmod(sudoers_file, 0o440)
def setup_logging(bench_path='.'):
if os.path.exists(os.path.join(bench_path, 'logs')):
@ -308,7 +310,12 @@ def get_git_version():
def check_git_for_shallow_clone():
from .config.common_site_config import get_config
if get_config(".").get('release_bench'):
config = get_config('.')
if config.get('release_bench'):
return False
if not config.get('shallow_clone'):
return False
git_version = get_git_version()
@ -319,9 +326,9 @@ def check_git_for_shallow_clone():
def get_cmd_output(cmd, cwd='.'):
try:
return subprocess.check_output(cmd, cwd=cwd, shell=True, stderr=open(os.devnull, 'wb')).strip()
except subprocess.CalledProcessError, e:
except subprocess.CalledProcessError as e:
if e.output:
print e.output
print(e.output)
raise
def restart_supervisor_processes(bench_path='.', web_workers=False):
@ -432,7 +439,7 @@ def drop_privileges(uid_name='nobody', gid_name='nogroup'):
os.setuid(running_uid)
# Ensure a very conservative umask
os.umask(022)
os.umask(0o22)
def fix_prod_setup_perms(bench_path='.', frappe_user=None):
from .config.common_site_config import get_config
@ -451,7 +458,7 @@ def fix_prod_setup_perms(bench_path='.', frappe_user=None):
frappe_user = get_config(bench_path).get('frappe_user')
if not frappe_user:
print "frappe user not set"
print("frappe user not set")
sys.exit(1)
for path in files:
@ -463,14 +470,14 @@ def fix_prod_setup_perms(bench_path='.', frappe_user=None):
def fix_file_perms():
for dir_path, dirs, files in os.walk('.'):
for _dir in dirs:
os.chmod(os.path.join(dir_path, _dir), 0755)
os.chmod(os.path.join(dir_path, _dir), 0o755)
for _file in files:
os.chmod(os.path.join(dir_path, _file), 0644)
os.chmod(os.path.join(dir_path, _file), 0o644)
bin_dir = './env/bin'
if os.path.exists(bin_dir):
for _file in os.listdir(bin_dir):
if not _file.startswith('activate'):
os.chmod(os.path.join(bin_dir, _file), 0755)
os.chmod(os.path.join(bin_dir, _file), 0o755)
def get_current_frappe_version(bench_path='.'):
from .app import get_current_frappe_version as fv
@ -498,7 +505,8 @@ def run_frappe_cmd(*args, **kwargs):
return_code = p.wait()
if return_code > 0:
raise CommandFailedError(args)
sys.exit(return_code)
#raise CommandFailedError(args)
def get_frappe_cmd_output(*args, **kwargs):
bench_path = kwargs.get('bench_path', '.')
@ -531,8 +539,8 @@ def post_upgrade(from_ver, to_ver, bench_path='.'):
from .config.supervisor import generate_supervisor_config
from .config.nginx import make_nginx_conf
conf = get_config(bench_path=bench_path)
print "-"*80
print "Your bench was upgraded to version {0}".format(to_ver)
print("-"*80)
print("Your bench was upgraded to version {0}".format(to_ver))
if conf.get('restart_supervisor_on_update'):
redis.generate_config(bench_path=bench_path)
@ -545,17 +553,17 @@ def post_upgrade(from_ver, to_ver, bench_path='.'):
if from_ver <= 5 and to_ver == 6:
setup_socketio(bench_path=bench_path)
print "As you have setup your bench for production, you will have to reload configuration for nginx and supervisor"
print "To complete the migration, please run the following commands"
print
print "sudo service nginx restart"
print "sudo supervisorctl reload"
print("As you have setup your bench for production, you will have to reload configuration for nginx and supervisor")
print("To complete the migration, please run the following commands")
print()
print("sudo service nginx restart")
print("sudo supervisorctl reload")
def update_translations_p(args):
try:
update_translations(*args)
except requests.exceptions.HTTPError:
print 'Download failed for', args[0], args[1]
print('Download failed for', args[0], args[1])
def download_translations_p():
pool = multiprocessing.Pool(4)
@ -592,7 +600,7 @@ def update_translations(app, lang):
f.write(chunk)
f.flush()
print 'downloaded for', app, lang
print('downloaded for', app, lang)
def download_chart_of_accounts():
charts_dir = os.path.join('apps', "erpnext", "erpnext", 'accounts', 'chart_of_accounts', "submitted")
@ -649,18 +657,18 @@ def validate_pillow_dependencies(bench_path, requirements):
distro = platform.linux_distribution()
distro_name = distro[0].lower()
if "centos" in distro_name or "fedora" in distro_name:
print "Please install these dependencies using the command:"
print "sudo yum install libtiff-devel libjpeg-devel libzip-devel freetype-devel lcms2-devel libwebp-devel tcl-devel tk-devel"
print("Please install these dependencies using the command:")
print("sudo yum install libtiff-devel libjpeg-devel libzip-devel freetype-devel lcms2-devel libwebp-devel tcl-devel tk-devel")
raise
elif "ubuntu" in distro_name or "elementary os" in distro_name or "debian" in distro_name:
print "Please install these dependencies using the command:"
print("Please install these dependencies using the command:")
if "ubuntu" in distro_name and distro[1]=="12.04":
print "sudo apt-get install -y libtiff4-dev libjpeg8-dev zlib1g-dev libfreetype6-dev liblcms2-dev libwebp-dev tcl8.5-dev tk8.5-dev python-tk"
print("sudo apt-get install -y libtiff4-dev libjpeg8-dev zlib1g-dev libfreetype6-dev liblcms2-dev libwebp-dev tcl8.5-dev tk8.5-dev python-tk")
else:
print "sudo apt-get install -y libtiff5-dev libjpeg8-dev zlib1g-dev libfreetype6-dev liblcms2-dev libwebp-dev tcl8.6-dev tk8.6-dev python-tk"
print("sudo apt-get install -y libtiff5-dev libjpeg8-dev zlib1g-dev libfreetype6-dev liblcms2-dev libwebp-dev tcl8.6-dev tk8.6-dev python-tk")
raise
@ -686,9 +694,18 @@ def set_git_remote_url(git_url, bench_path='.'):
app = git_url.rsplit('/', 1)[1].rsplit('.', 1)[0]
if app not in bench.app.get_apps(bench_path):
print "No app named {0}".format(app)
print("No app named {0}".format(app))
sys.exit(1)
app_dir = bench.app.get_repo_dir(app, bench_path=bench_path)
if os.path.exists(os.path.join(app_dir, '.git')):
exec_cmd("git remote set-url upstream {}".format(git_url), cwd=app_dir)
exec_cmd("git remote set-url upstream {}".format(git_url), cwd=app_dir)
def run_playbook(playbook_name, extra_vars=None):
if not find_executable('ansible'):
print("Ansible is needed to run this command, please install it using 'pip install ansible'")
sys.exit(1)
args = ['ansible-playbook', '-c', 'local', playbook_name]
if extra_vars:
args.extend(['-e', json.dumps(extra_vars)])
subprocess.check_call(args, cwd=os.path.join(os.path.dirname(bench.__path__[0]), 'playbooks'))

22
playbooks/develop/centos.yml
View File

@ -5,7 +5,6 @@
bench_path: "/home/{{ ansible_user_id }}/frappe-bench"
mysql_config_template: "templates/simple_mariadb_config.cnf"
mysql_conf_dir: /etc/my.cnf.d/
wkhtmltopdf_version: 0.12.2.1
tasks:
@ -41,19 +40,24 @@
- libtiff-devel
- tcl-devel
- tk-devel
# Python LDAP
- openldap-devel
become: yes
become_user: root
- name: Get nodejs 6.x bash script
get_url:
url: 'https://rpm.nodesource.com/setup_6.x'
dest: '/tmp/setup_6.x'
mode: 0644
- name: Import Node source RPM key
rpm_key:
key: https://rpm.nodesource.com/pub/el/NODESOURCE-GPG-SIGNING-KEY-EL
state: present
become: yes
become_user: root
- name: Run nodejs 6.x bash script
command: /bin/bash /tmp/setup_6.x
- name: Add Node Repo
yum:
name: 'https://rpm.nodesource.com/pub_6.x/el/{{ ansible_distribution_major_version }}/{{ ansible_architecture }}/nodesource-release-el{{ ansible_distribution_major_version }}-1.noarch.rpm'
state: present
become: yes
become_user: root
@ -68,7 +72,7 @@
- include: includes/mariadb_centos.yml
# install WKHTMLtoPDF
- include: includes/wkhtmltopdf_centos.yml
- include: includes/wkhtmltopdf.yml
# setup MariaDB
- include: includes/setup_mariadb.yml

31
playbooks/develop/debian.yml
View File

@ -5,7 +5,6 @@
bench_path: "/home/{{ ansible_user_id }}/frappe-bench"
mysql_config_template: "templates/simple_mariadb_config.cnf"
mysql_conf_dir: /etc/mysql/conf.d/
wkhtmltopdf_version: 0.12.2.1
tasks:
@ -49,10 +48,17 @@
- libwebp-dev
- python-tk
# Ensure apt-transport-https
- apt-transport-https
# Python LDAP
- libsasl2-dev
- libldap2-dev
become: yes
become_user: root
- name: install pillow prerequisites for Debian < 8
- name: install pillow prerequisites for Debian < 8
apt: pkg={{ item }} state=present
with_items:
- libjpeg8-dev
@ -74,16 +80,19 @@
become: yes
become_user: root
- name: Get nodejs 6.x bash script
get_url:
url: 'https://deb.nodesource.com/setup_6.x'
dest: '/tmp/setup_6.x'
mode: 0644
- name: Add apt key for node repo
apt_key:
url: https://keyserver.ubuntu.com/pks/lookup?op=get&fingerprint=on&search=0x1655A0AB68576280
id: "68576280"
state: present
become: yes
become_user: root
- name: Run nodejs bash script
command: /bin/bash /tmp/setup_6.x
- name: Add repo
apt_repository:
repo: "deb [arch=amd64,i386] https://deb.nodesource.com/node_6.x {{ ansible_distribution_release }} main"
state: present
update_cache: yes
become: yes
become_user: root
@ -95,12 +104,12 @@
force: yes
become: yes
become_user: root
# install MariaDB
- include: includes/mariadb_debian.yml
# install WKHTMLtoPDF
- include: includes/wkhtmltopdf_ubuntu_debian.yml
- include: includes/wkhtmltopdf.yml
# setup MariaDB
- include: includes/setup_mariadb.yml

19
playbooks/develop/includes/wkhtmltopdf.yml Normal file
View File

@ -0,0 +1,19 @@
---
- name: download wkthmltox linux
get_url: url=http://download.gna.org/wkhtmltopdf/0.12/0.12.3/wkhtmltox-0.12.3_linux-generic-{{ "amd64" if ansible_architecture == "x86_64" else "i386"}}.tar.xz dest=/tmp/wkhtmltox.tar.xz
- name: Creates directory
file: path=/tmp/wkhtmltox state=directory
- name: unarchive wkhtmltopdf
unarchive: src=/tmp/wkhtmltox.tar.xz dest=/tmp
- name: copy to /usr/local/bin
copy: src="/tmp/wkhtmltox/bin/wkhtmltopdf" dest="/usr/local/bin/wkhtmltopdf"
become: true
become_user: root
- name: make wkhtmltopdf executable
file: path=/usr/local/bin/wkhtmltopdf mode="o+x"
become: true
become_user: root

12
playbooks/develop/includes/wkhtmltopdf_centos.yml
View File

@ -1,12 +0,0 @@
---
- name: Download wkhtmltopdf
get_url:
url=http://download.gna.org/wkhtmltopdf/0.12/{{ wkhtmltopdf_version }}/wkhtmltox-{{ wkhtmltopdf_version }}_linux-centos{{ ansible_lsb.major_release }}-{{ "amd64" if ansible_architecture == "x86_64" else "i386"}}.rpm
dest="/tmp/"
become: yes
become_user: root
- name: Install wkhtmltopdf deb
yum: name=/tmp/wkhtmltox-{{ wkhtmltopdf_version }}_linux-centos{{ ansible_lsb.major_release }}-{{ "amd64" if ansible_architecture == "x86_64" else "i386"}}.rpm state=present
become: yes
become_user: root

28
playbooks/develop/includes/wkhtmltopdf_ubuntu_debian.yml
View File

@ -1,28 +0,0 @@
---
- name: Download wkhtmltopdf
get_url:
url=http://download.gna.org/wkhtmltopdf/0.12/{{ wkhtmltopdf_version }}/wkhtmltox-{{ wkhtmltopdf_version }}_linux-{{ ansible_distribution_release }}-{{ "amd64" if ansible_architecture == "x86_64" else "i386"}}.deb
dest="/tmp/"
become: yes
become_user: root
when: ansible_distribution_release in ("jessie", "precise", "trusty", "wheezy")
- name: Install wkhtmltopdf deb
apt: deb=/tmp/wkhtmltox-{{ wkhtmltopdf_version }}_linux-{{ ansible_distribution_release }}-{{ "amd64" if ansible_architecture == "x86_64" else "i386"}}.deb
become: yes
become_user: root
when: ansible_distribution_release in ("jessie", "precise", "trusty", "wheezy")
- name: Download wkhtmltopdf for Ubuntu Wily
get_url:
url=http://download.gna.org/wkhtmltopdf/0.12/{{ wkhtmltopdf_version }}/wkhtmltox-{{ wkhtmltopdf_version }}_linux-trusty-{{ "amd64" if ansible_architecture == "x86_64" else "i386"}}.deb
dest="/tmp/"
become: yes
become_user: root
when: ansible_distribution_release=="wily"
- name: Install wkhtmltopdf deb for Ubuntu Wily
apt: deb=/tmp/wkhtmltox-{{ wkhtmltopdf_version }}_linux-trusty-{{ "amd64" if ansible_architecture == "x86_64" else "i386"}}.deb
become: yes
become_user: root
when: ansible_distribution_release=="wily"

27
playbooks/develop/ubuntu.yml
View File

@ -5,7 +5,6 @@
bench_path: "/home/{{ ansible_user_id }}/frappe-bench"
mysql_config_template: "templates/simple_mariadb_config.cnf"
mysql_conf_dir: /etc/mysql/conf.d/
wkhtmltopdf_version: 0.12.2.1
tasks:
@ -34,6 +33,13 @@
- libwebp-dev
- python-tk
# Ensure apt-transport-https
- apt-transport-https
# Python LDAP
- libsasl2-dev
- libldap2-dev
become: yes
become_user: root
@ -57,16 +63,19 @@
become: yes
become_user: root
- name: Get nodejs 6.x bash script
get_url:
url: 'https://deb.nodesource.com/setup_6.x'
dest: '/tmp/setup_6.x'
mode: 0644
- name: Add apt key for node repo
apt_key:
url: https://keyserver.ubuntu.com/pks/lookup?op=get&fingerprint=on&search=0x1655A0AB68576280
id: "68576280"
state: present
become: yes
become_user: root
- name: Run nodejs bash script
command: /bin/bash /tmp/setup_6.x
- name: Add repo
apt_repository:
repo: "deb [arch=amd64,i386] https://deb.nodesource.com/node_6.x {{ ansible_distribution_release }} main"
state: present
register: node_repo
become: yes
become_user: root
@ -83,7 +92,7 @@
- include: includes/mariadb_ubuntu.yml
# install WKHTMLtoPDF
- include: includes/wkhtmltopdf_ubuntu_debian.yml
- include: includes/wkhtmltopdf.yml
# setup MariaDB
- include: includes/setup_mariadb.yml

33
playbooks/install.py
View File

@ -30,7 +30,7 @@ def install_bench(args):
})
if not success:
print 'Could not install pre-requisites. Please check for errors or install them manually.'
print('Could not install pre-requisites. Please check for errors or install them manually.')
return
# secure pip installation
@ -75,7 +75,7 @@ def install_bench(args):
if args.production:
args.user = 'frappe'
elif os.environ.has_key('SUDO_USER'):
elif 'SUDO_USER' in os.environ:
args.user = os.environ['SUDO_USER']
else:
@ -122,9 +122,9 @@ def check_distribution_compatibility():
if float(dist_version) in supported_dists[dist_name]:
return
print "Sorry, the installer doesn't support {0} {1}. Aborting installation!".format(dist_name, dist_version)
print("Sorry, the installer doesn't support {0} {1}. Aborting installation!".format(dist_name, dist_version))
if dist_name in supported_dists:
print "Install on {0} {1} instead".format(dist_name, supported_dists[dist_name][-1])
print("Install on {0} {1} instead".format(dist_name, supported_dists[dist_name][-1]))
sys.exit(1)
def get_distribution_info():
@ -135,14 +135,14 @@ def get_distribution_info():
elif platform.system() == "Darwin":
current_dist = platform.mac_ver()
return "macos", current_dist[0].rsplit('.', 1)[0]
def install_python27():
version = (sys.version_info[0], sys.version_info[1])
if version == (2, 7):
return
print 'Installing Python 2.7'
print('Installing Python 2.7')
# install python 2.7
success = run_os_command({
@ -210,9 +210,9 @@ def clone_bench_repo(args):
def run_os_command(command_map):
'''command_map is a dictionary of {'executable': command}. For ex. {'apt-get': 'sudo apt-get install -y python2.7'} '''
success = True
for executable, commands in command_map.items():
for executable, commands in list(command_map.items()):
if find_executable(executable):
if isinstance(commands, basestring):
if isinstance(commands, str):
commands = [commands]
for command in commands:
@ -245,7 +245,7 @@ def get_passwords(ignore_prompt=False):
# admin password
if not admin_password:
admin_password = getpass.unix_getpass(prompt='Please enter Administrator password: ')
admin_password = getpass.unix_getpass(prompt='Please enter the default Administrator user password: ')
conf_admin_passswd = getpass.unix_getpass(prompt='Re-enter Administrator password: ')
if admin_password != conf_admin_passswd:
@ -256,16 +256,25 @@ def get_passwords(ignore_prompt=False):
else:
mysql_root_password = admin_password = 'travis'
return {
passwords = {
'mysql_root_password': mysql_root_password,
'admin_password': admin_password
}
if not ignore_prompt:
passwords_file_path = os.path.join(os.path.expanduser('~'), 'passwords.txt')
with open(passwords_file_path, 'w') as f:
json.dump(passwords, f, indent=1)
print('Passwords saved at ~/passwords.txt')
return passwords
def get_extra_vars_json(extra_args):
# We need to pass production as extra_vars to the playbook to execute conditionals in the
# playbook. Extra variables can passed as json or key=value pair. Here, we will use JSON.
json_path = os.path.join('/tmp', 'extra_vars.json')
extra_vars = dict(extra_args.items())
extra_vars = dict(list(extra_args.items()))
with open(json_path, mode='w') as j:
json.dump(extra_vars, j, indent=1, sort_keys=True)
@ -340,3 +349,5 @@ if __name__ == '__main__':
args = parse_commandline_args()
install_bench(args)
print('''Frappe/ERPNext has been successfully installed!''')

14
playbooks/production/change_ssh_port.yml Executable file
View File

@ -0,0 +1,14 @@
- name: Change ssh port
gather_facts: false
hosts: localhost
user: root
tasks:
- name: change sshd config
lineinfile: >
dest=/etc/ssh/sshd_config
regexp="^Port"
line="Port {{ ssh_port }}"
state=present
- name: restart ssh
service: name=sshd state=reloaded

22
playbooks/production/includes/setup_prod_env.yml
View File

@ -78,6 +78,28 @@
become: yes
become_user: root
######################################################
# Set InnoDB Buffer Pool size to half of total RAM
- name: Set InnoDB buffer pool
lineinfile: >
dest=/etc/my.cnf.d/frappe.cnf
regexp="^\[mysqld\]$"
line="[mysqld]\ninnodb_buffer_pool_size={{ (ansible_memtotal_mb/2)|round|int }}M"
state=present
when: ansible_distribution == 'CentOS'
become: yes
become_user: root
- name: Set InnoDB buffer pool
lineinfile: >
dest=/etc/mysql/conf.d/frappe.cnf
regexp="^\[mysqld\]$"
line="[mysqld]\ninnodb_buffer_pool_size={{ (ansible_memtotal_mb/2)|round|int }}M"
state=present
when: ansible_distribution == 'Ubuntu' or ansible_distribution == 'Debian'
become: yes
become_user: root
####################################################
# Enable nginx, mysql, redis and supevisord services
- name: Enable nginx, mysql, and redis

43
playbooks/production/setup_firewall.yml Executable file
View File

@ -0,0 +1,43 @@
- name: Setup Firewall
user: root
hosts: localhost
tasks:
# For CentOS
- name: Install firewalld
yum: name=firewalld state=present
when: ansible_distribution == 'CentOS'
- name: Enable Firewall
service: name=firewalld state=started enabled=yes
when: ansible_distribution == 'CentOS'
- name: Add firewall rules
firewalld: port={{ item }}/tcp permanent=true state=enabled
with_items:
- 80
- 443
- 22
when: ansible_distribution == 'CentOS'
- name: Restart Firewall
service: name=firewalld state=restarted enabled=yes
when: ansible_distribution == 'CentOS'
# For Ubuntu / Debian
- name: Install ufw
apt: name=ufw state=present
when: ansible_distribution == 'Ubuntu' or ansible_distribution == 'Debian'
- name: Enable Firewall
ufw: state=enabled policy=deny
when: ansible_distribution == 'Ubuntu' or ansible_distribution == 'Debian'
- name: Add firewall rules
ufw: rule=allow proto=tcp port={{ item }}
with_items:
- 80
- 443
- 22
when: ansible_distribution == 'Ubuntu' or ansible_distribution == 'Debian'

2
vm/ansible/roles/wkhtmltopdf/defaults/main.yml
View File

@ -1 +1 @@
wkhtmltopdf_version: 0.12.2.1
wkhtmltopdf_version: 0.12.3

19
vm/ansible/roles/wkhtmltopdf/tasks/main.yml
View File

@ -8,10 +8,6 @@
- xorg-x11-fonts-Type1
when: ansible_os_family == 'RedHat'
- name: Install wkhtmltopdf rpm
yum: name=http://download.gna.org/wkhtmltopdf/0.12/{{ wkhtmltopdf_version }}/wkhtmltox-{{ wkhtmltopdf_version }}_linux-centos{{ ansible_distribution_major_version }}-{{ "amd64" if ansible_architecture == "x86_64" else "i386"}}.rpm
when: ansible_os_family == 'RedHat'
- name: install base fonts
apt: name={{ item }} state=present
with_items:
@ -21,12 +17,11 @@
- xfonts-base
when: ansible_os_family == 'Debian'
- name: Download wkhtmltopdf
get_url:
url=http://download.gna.org/wkhtmltopdf/0.12/{{ wkhtmltopdf_version }}/wkhtmltox-{{ wkhtmltopdf_version }}_linux-{{ ansible_distribution_release }}-{{ "amd64" if ansible_architecture == "x86_64" else "i386"}}.deb
dest="/tmp/"
when: ansible_os_family == 'Debian'
- name: download wkthmltox linux
get_url: url=http://download.gna.org/wkhtmltopdf/0.12/0.12.3/wkhtmltox-0.12.3_linux-generic-{{ "amd64" if ansible_architecture == "x86_64" else "i386"}}.tar.xz dest=/tmp/wkhtmltox.tar.xz
- name: Install wkhtmltopdf deb
apt: deb=/tmp/wkhtmltox-{{ wkhtmltopdf_version }}_linux-{{ ansible_distribution_release }}-{{ "amd64" if ansible_architecture == "x86_64" else "i386"}}.deb
when: ansible_os_family == 'Debian'
- name: unarchive wkhtmltopdf
unarchive: src=/tmp/wkhtmltox.tar.xz dest=/tmp/wkhtmltox
- name: move to /usr/local/bin
command: creates="/usr/local/bin/wkhtmltopdf" mv /tmp/wkhtmltox/bin/wkhtmltopdf /usr/local/bin/wkhtmltopdf