telegram-bot-bash/4_expert.md at a23370645f950920d4bd71764a4c1c1c767e2996

mirror of https://github.com/octoleo/telegram-bot-bash.git synced 2024-11-25 00:37:34 +00:00

Kay Marquardt (Gnadelwartz) 36d8604cde Bashbot Version 1.51

2021-06-03 14:21:40 +02:00

Expert Use

Handling UTF-8 character sets

UTF-8 is a variable length encoding of Unicode. UTF-8 is recommended as the default encoding in JSON, XML and HTML, also Telegram make use of it.

The first 128 characters are regular ASCII, so it's a superset of and compatible with ASCII environments. The next 1,920 characters need two bytes for encoding and covers almost all Latin alphabets, also Greek, Cyrillic, Hebrew, Arabic and more. See Wikipedia for more details.

Setting up your Environment

In general bash and GNU utitities are UTF-8 aware if you to setup your environment and your scripts accordingly (locale setting):

Your Terminal and Editor must support UTF-8: Set Terminal and Editor locale to UTF-8, eg. in Settings/Configuration select UTF-8 (Unicode) as Charset.
Set Shell locale environment to UTF-8 in your .profile and your scripts. The usual settings are:

export 'LC_ALL=C.UTF-8'
export 'LANG=C.UTF-8'
export 'LANGUAGE=C.UTF-8'

If you use other languages, eg. german or US english, change the shell settings to:

export 'LC_ALL=de_DE.UTF-8'
export 'LANG=de_DE.UTF-8'
export 'LANGUAGE=de_DE.UTF-8'

export 'LC_ALL=en_US.UTF-8'
export 'LANG=de_en_US.UTF-8'
export 'LANGUAGE=en_US.UTF-8'

make sure your bot scripts use the correct settings, eg. include the lines above at the beginning of your scripts

Known locale pitfalls

Missing C locale

Even required by POSIX standard some systems (e.g. Manjaro Linux) has C and C.UTF-8 locale not installed. If bashbot display a warning about missing locale you must install C and C.UTF-8 locale.

If you don't know what locales are installed on your sytsem use locale -a to display them. Gentoo Wiki.

Character classes

In ASCII times it was clear [:lower:] and [a-z] means ONLY the lowercase letters [abcd...xyz]. With the introduction of locales, character classes and ranges now contain all characters fitting the class definition.

This means with a Latin UTF-8 locale [:lower:] and [a-z] contains also e.g. á ø ü etc, see Unicode Latin lowercase letters

If that's ok for your script you're fine, but many scripts rely on the idea of ASCII ranges and may produce undesired results.

# try with different locales ...
# new bash to not change your current locale!
bash
lower="abcö"

echo "$LC_ALL $LC_COLLATE"
[[ "$lower" =~ ^[a-z]+$ ]] && echo "Ups, $lower is all lower case!" || echo "OK, not lower case"

LC_ALL="en_US.UTF-8"
[[ "$lower" =~ ^[a-z]+$ ]] && echo "Ups, $lower is all lower case!" || echo "OK, not lower case"

LC_ALL="C"
[[ "$lower" =~ ^[a-z]+$ ]] && echo "Ups, $lower is all lower case!" || echo "OK, not lower case"

There are three solutions:

list exactly the characters you want: [abcd...]
instruct bash to use C locale for ranges: shopt -s "globasciiranges"
use LC_COLLATE to change behavior of all programs: export LC_COLLATE=C

To work independent of language and bash settings bashbot uses solution 1.: Own "ranges" if an exact match is mandatory:

azazaz='abcdefghijklmnopqrstuvwxyz'	# a-z   :lower:
AZAZAZ='ABCDEFGHIJKLMNOPQRSTUVWXYZ'	# A-Z   :upper:
o9o9o9='0123456789'			# 0-9   :digit:
azAZaz="${azazaz}${AZAZAZ}"	# a-zA-Z	:alpha:
azAZo9="${azAZaz}${o9o9o9}"	# a-zA-z0-9	:alnum:

# e.g. characters allowed for key in key/value pairs
JSSH_KEYOK="[-${azAZo9},._]"

Bashbot UTF-8 Support

Bashbot handles all messages transparently, regardless of the charset in use. The only exception is when converting from JSON data to strings.

Telegram use JSON to send / receive data. JSON encodes strings as follow: Characters not ASCII (>127) are escaped as sequences of \uxxxx to be regular ASCII. In addition multibyte characters, e.g. Emoticons or Arabic characters, are send in double byte UTF-16 notation. The Emoticons 😁 😘 ❤️ 😊 👍 are encoded as: \uD83D\uDE01 \uD83D\uDE18 \u2764\uFE0F \uD83D\uDE0A \uD83D\uDC4D

This "mixed" JSON encoding needs special handling and can not decoded from echo -e or printf '%s\\n'

Bashbot uses an internal, pure bash implementation which is well tested now, even there may some corner cases*.

Run as other user or system service

Bashbot is designed to run manually by the user who installed it. Nevertheless it's possible to run it by an other user-ID, as a system service or scheduled from cron. This is recommended if you want to bashbot run as a service.

Setup the environment for the user you want to run bashbot and enter desired username, e.g. nobody :

sudo ./bashbot.sh init

Edit the file bashbot.rc and change the following lines to fit your configuration:

#######################
# Configuration Section

# edit the next line to fit the user you want to run bashbot, e.g. nobody:
runas="nobody" 

# uncomment one of the following lines 
# runcmd="su $runas -s /bin/bash -c "      # runasuser with su
# runcmd="runuser $runas -s /bin/bash -c " # runasuser with runuser

# edit the values of the following lines to fit your config:
start="/usr/local/telegram-bot-bash/bashbot.sh"	# location of your bashbot.sh script
name=''   # your bot name as given to botfather, e.g. mysomething_bot

# END Configuration
#######################

From now on use 'bashbot.rc' to manage your bot:

sudo ./bashbot.rc start

Type ps -ef | grep bashbot to verify your Bot is running as the desired user.

If your Bot is started by 'bashbot.rc', you must use 'bashbot.rc' also to manage your Bot! The following commands are available:

sudo ./bashbot.rc start
sudo ./bashbot.rc stop
sudo ./bashbot.rc status
sudo ./bashbot.rc suspendback
sudo ./bashbot.rc resumeback
sudo ./bashbot.rc killback

To change back the environment to your user-ID run sudo ./bashbot.sh init again and enter your user name.

To use bashbot as a system service include a working bashbot.rc in your init system (systemd, /etc/init.d).

Schedule bashbot from Cron

An example crontab is provided in examples/bashbot.cron.

If you are running bashbot with your user-ID, copy the examples lines to your crontab and remove username nobody.
if you run bashbot as an other user or a system service edit examples/bashbot.cron to fit your needs and replace username nobody with the username you want to run bashbot. Copy the modified file to /etc/cron.d/bashbot

Use bashbot from CLI and scripts

You can use bashbot to send messages, locations, venues, pictures etc. from command line and scripts by sourcing it:

usage: . bashbot.sh source

Before sourcing 'bashbot.sh' for interactive and script use, you should export and set BASHBOT_HOME to bashbots installation dir, e.g. '/usr/local/telegram-bot-bash'. see Bashbot Environment

Note: If you don't set BASHBOT_HOME bashbot will use the actual directory as NEW home directory which means it will create all needed files and ask for bot token and botadmin if you are not in the real bot home!

Examples:

# if you are in the bashbot directory
.  bashbot.sh source

# same, but more readable in scripts
source ./bashbot.sh source

# use bashbot config in BASHBOT_HOME from any directory
export BASHBOT_HOME=/usr/local/telegram-bot-bash
source ${BASHBOT_HOME}/bashbot.sh source

# use / create new config in current directory
unset  BASHBOT_HOME
source /path/to/bashbot.sh source

Environment variable exported from bashbot

If you have sourced 'bashbot.sh' you have the following bashot internal variables available:

COMMANDS	# default: ./commands.sh"
MODULEDIR	# default: ./modules"
BOTACL		# default: ./botacl"
TMPDIR		# default: ./data-bot-bash"
COUNTFILE	# default: ./count" (jsonDB file)

BOTTOKEN	# your token read from bot config
URL		# telegram api URL - default: https://api.telegram.org/bot${BOTTOKEN}"

Interactive use

For testing your setup or sending messages yourself its possible to use bashbot functions from command line:

# are we running bash?
echo $SHELL
/bin/bash

# source bashbot.sh WITHOUT BASHBOT_HOME set
source ./bashbot.sh source

# output bashbot internal variables
echo $COMMANDS $MODULEDIR  $BOTACL $TMPDIR $COUNTFILE
./commands.sh ./modules  ./botacl ./data-bot-bash ./count


# source bashbot.sh WITH BASHBOT_HOME set
export BASHBOT_HOME=/usr/local/telegram-bot-bash
source ./bashbot.sh source

# output bashbot internal variables
echo $COMMANDS $MODULEDIR $BOTACL $TMPDIR $COUNTFILE
/usr/local/telegram-bot-bash/commands.sh /usr/local/telegram-bot-bash/modules 
/usr/local/telegram-bot-bash/botacl /usr/local/telegram-bot-bash/data-bot-bash
/usr/local/telegram-bot-bash/count

After sourcing you can use bashbot functions to send Messages, Locations, Pictures etc. to any Telegram User or Chat you are in. See Send Messages.

Examples: You can test this by sending messages to yourself:

# first Hello World
send_normal_message "$(getConfigKey "botadmin")"  "Hello World! This is my first message"

# now with some markdown and HTML
send_markdown_message 	"$(getConfigKey "botadmin")"  '*Hello World!* _This is my first markdown message_'
send_html_message	"$(getConfigKey "botadmin")"  '<b>Hello World!</b> <em>This is my first HTML message</em>'
send_keyboard "$(getConfigKey "botadmin")"  'Do you like it?' '[ "Yep" , "No" ]'

Now something more useful ...

# sending output from system commands:
send_normal_message	"$(getConfigKey "botadmin")"  "$(date)"

send_normal_message	"$(getConfigKey "botadmin")"  "$(uptime)"

send_normal_message       "$(getConfigKey "botadmin")"  '`'$(free)'`'

# same but markdown style 'code' (monospaced)
send_markdown_message	"$(getConfigKey "botadmin")"  "\`$(free)\`"

Bashbot environment

This section describe how you can customize bashbot to your needs by setting environment variables.

Change file locations

In standard setup bashbot is self containing, this means you can place 'telegram-bot-bash' any location and run it from there. All files - program, config, data etc - will reside in 'telegram-bot-bash'.

If you want to have other locations for config, data etc, define and export the following environment variables. Note: all specified directories and files must exist or running 'bashbot.sh' will fail.

BASHBOT_ETC

Location of the files commands.sh, mycommands.sh, botconfig.jssh, botacl ...

  unset  BASHBOT_ETC     # keep in telegram-bot-bash (default)
  export BASHBOT_ETC ""  # keep in telegram-bot-bash

  export BASHBOT_ETC "/etc/bashbot"  # Unix-like config location

  export BASHBOT_ETC "/etc/bashbot/bot1"  # multibot configuration bot 1
  export BASHBOT_ETC "/etc/bashbot/bot2"  # multibot configuration bot 2

e.g. /etc/bashbot

BASHBOT_VAR

Location of runtime data data-bot-bash, count.jssh

  unset  BASHBOT_VAR     # keep in telegram-bot-bash (default)
  export BASHBOT_VAR ""  # keep in telegram-bot-bash

  export BASHBOT_VAR "/var/spool/bashbot"  # Unix-like config location

  export BASHBOT_VAR "/var/spool/bashbot/bot1"  # multibot configuration bot 1
  export BASHBOT_VAR "/var/spool/bashbot/bot2"  # multibot configuration bot 2

BASHBOT_JSONSH

Full path to JSON.sh script, default: './JSON.sh/JSON.sh', must end with '/JSON.sh'.

  unset  BASHBOT_JSONSH     # telegram-bot-bash/JSON.sh/JSON.sh (default)
  export BASHBOT_JSONSH ""  # telegram-bot-bash/JSON.sh/JSON.sh

  export BASHBOT_JSONSH "/usr/local/bin/JSON.sh"  # installed in /usr/local/bin

BASHBOT_HOME

Set bashbot home directory, where bashot will look for additional files. If BASHBOT_ETC, BASHBOT_VAR or BASHBOT_JSONSH are set the have precedence over BASHBOT_HOME.

This is also useful if you want to force bashbot to always use full pathnames instead of relative ones.

  unset  BASHBOT_HOME     # autodetection (default)
  export BASHBOT_HOME ""  # autodetection

  export BASHBOT_HOME "/usr/local/telegram-bot-bash"	# Unix-like location
  export BASHBOT_HOME "/usr/local/bin"	# Note: you MUST set ETC, VAR and JSONSH to other locations to make this work!

Change config values

BASHBOT_URL

Uses given URL instead of official Telegram API URL, useful if you have your own telegram server or for testing.

  unset  BASHBOT_URL       # use Telegram URL https://api.telegram.org/bot<token> (default)

  export BASHBOT_URL ""    # use use Telegram https://api.telegram.org/bot<token>

  export BASHBOT_URL "https://my.url.com/bot" # use your URL https://my.url.com/bot<token>

BASHBOT_TOKEN

If BASHBOT_TOKEN is set, bashbot assumes you know what you are doing and skips environment validation and uses the value of BASHBOT_TOKEN as bot token.

I recommend to run 'bashbot.sh init' at least one time without BASHBOT_TOKEN set to validate and setup the environment. Afterwards you can delete the token file and provide the bot token in BASHBOT_TOKEN.

BASHBOT_CURL_ARGS

The value of BASHBOT_CURL_ARGS is passed to every curl execution.

  # use socks gateway on localhost
  export BASHBOT_CURL_ARGS="--socks5-hostname localhost"

BASHBOT_CURL

If BASHBOT_CURL is not set your systems default curl is used. If you want to use an alternative curl executable set BASHBOT_CURL to point to it.

  # use curl from /usr/local/bin
  export BASHBOT_CURL="/usr/local/bin/mycurl"

BASHBOT_WGET

Bashbot uses curl to communicate with telegram server. if curl is not available wget is used. If 'BASHBOT_WGET' is set to any value (not undefined or not empty) wget is used even is curl is available.

  unset  BASHBOT_WGET       # use curl (default)
  export BASHBOT_WGET ""    # use curl 

  export BASHBOT_WGET "yes" # use wget
  export BASHBOT_WGET "no"  # use wget!

BASHBOT_TIMEOUT

Bashbot uses a default timeout of 20 seconds for curl and wget. If you want a different timeout, set BASHBOT_TIMEOUT to a numeric value between 1 and 999. Any non numeric or negative value is ignored.

  # set timeout to 100 seconds
  export BASHBOT_TIMEOUT="100"

  # 100s is not a numbers
  export BASHBOT_TIMEOUT="100s" # wrong, default timeout is used

  # -100 is not between 1 and 999s
  export BASHBOT_TIMEOUT="-100" # wrong, default timeout is used

BASHBOT_SLEEP

Instead of polling permanently or with a fixed delay, bashbot offers a simple adaptive polling. If messages are received bashbot polls with no delay. If no messages are available bashbot add 100ms delay for every poll until the maximum of BASHBOT_SLEEP ms.

  unset  BASHBOT_SLEEP       # 5000ms (default)
  export BASHBOT_SLEEP ""    # 5000ms 

  export BASHBOT_SLEEP "1000"     # 1s maximum sleep 
  export BASHBOT_SLEEP "10000"    # 10s maximum sleep
  export BASHBOT_SLEEP "1"        # values < 1000 disables sleep (not recommended)

Tested configs as of v0.90 release

simple Unix like config, for one bot. bashbot is installed in '/usr/local/telegram-bot-bash'

  # Note: all dirs and files must exist!
  export BASHBOT_ETC "/etc/bashbot"
  export BASHBOT_VAR "/var/spool/bashbot"

  /usr/local/telegram-bot-bash/bashbot.sh start

Unix like config for one bot. bashbot.sh is installed in '/usr/bin'

  # Note: all dirs and files must exist!
  export BASHBOT_ETC "/etc/bashbot"
  export BASHBOT_VAR "/var/spool/bashbot"
  export BASHBOT_JSONSH "/var/spool/bashbot"

  /usr/local/bin/bashbot.sh start

simple multibot config, everything is kept inside 'telegram-bot-bash' dir

  # config for running Bot 1
  # Note: all dirs and files must exist!
  export BASHBOT_ETC "./mybot1"
  export BASHBOT_VAR "./mybot1"

  /usr/local/telegram-bot-bash/bashbot.sh start

  # config for running Bot 2
  # Note: all dirs and files must exist!
  export BASHBOT_ETC "./mybot2"
  export BASHBOT_VAR "./mybot2"

  /usr/local/telegram-bot-bash/bashbot.sh start

16 KiB

Raw Blame History

Home

Expert Use

Handling UTF-8 character sets

Setting up your Environment

Known locale pitfalls

Missing C locale

Character classes

Bashbot UTF-8 Support

Run as other user or system service

Schedule bashbot from Cron

Use bashbot from CLI and scripts

Environment variable exported from bashbot

Interactive use

Bashbot environment

Change file locations

BASHBOT_ETC

BASHBOT_VAR

BASHBOT_JSONSH

BASHBOT_HOME

Change config values

BASHBOT_URL

BASHBOT_TOKEN

BASHBOT_CURL_ARGS

BASHBOT_CURL

BASHBOT_WGET

BASHBOT_TIMEOUT

BASHBOT_SLEEP

Tested configs as of v0.90 release

simple Unix like config, for one bot. bashbot is installed in '/usr/local/telegram-bot-bash'

Unix like config for one bot. bashbot.sh is installed in '/usr/bin'

simple multibot config, everything is kept inside 'telegram-bot-bash' dir

Prev Advanced Use

Next Best Practice

$$VERSION$$ v1.51-0-g6e66a28

16 KiB Raw Blame History

Home

Expert Use

Handling UTF-8 character sets

Setting up your Environment

Known locale pitfalls

Missing C locale

Character classes

Bashbot UTF-8 Support

Run as other user or system service

Schedule bashbot from Cron

Use bashbot from CLI and scripts

Environment variable exported from bashbot

Interactive use

Bashbot environment

Change file locations

BASHBOT_ETC

BASHBOT_VAR

BASHBOT_JSONSH

BASHBOT_HOME

Change config values

BASHBOT_URL

BASHBOT_TOKEN

BASHBOT_CURL_ARGS

BASHBOT_CURL

BASHBOT_WGET

BASHBOT_TIMEOUT

BASHBOT_SLEEP

Tested configs as of v0.90 release

simple Unix like config, for one bot. bashbot is installed in '/usr/local/telegram-bot-bash'

Unix like config for one bot. bashbot.sh is installed in '/usr/bin'

simple multibot config, everything is kept inside 'telegram-bot-bash' dir

Prev Advanced Use

Next Best Practice

$$VERSION$$ v1.51-0-g6e66a28

16 KiB

Raw Blame History