Difference between revisions of "Bash notes"

From Noah.org
Jump to navigationJump to search
 
(41 intermediate revisions by the same user not shown)
Line 1: Line 1:
 
[[Category:Engineering]]
 
[[Category:Engineering]]
 +
[[Category:Shell]]
  
 +
= Bash, mostly =
  
== A better, more flexible history logger using rsyslog ==
 
  
 +
== Bash function to return formatted date in days past for both GNU Linux and MacOS OSX ==
 +
 +
Pass the number of days in the past for which you want the date. If you pass nothing it will assume 0 days.
 +
<pre>
 +
date_days_past () { days_past=${1:-0}; if ! date -v-${days_past}d +%Y%m%d 2>/dev/null; then date --date="-${days_past} day" +%Y%m%d; fi }
 +
</pre>
 +
 +
== script intended to be sourced only ==
 +
 +
If you have a script that is intended to be sourced instead of executed directly you can add an error message in case the script is executed directly. ... You could also use the trick of exec'ing a subshell that sources the file, but that's kind of ugly.
 +
<pre>
 +
#!/bin/echo "Source this file. Do no run it."
 +
</pre>
 +
 +
== Replicate SUID permissions ==
 +
 +
I had a filesystem where I lost all SUID permissions on executables. I mounted the filesystem under a different server and used the following to restore the SUID bit on all executables that needed it (especially '''sudo'''). I mounted the filesystem to be repaired under '''/mnt'''. I assume it has pretty much the same layout as the reference filesystem.
 +
<pre>
 +
for fn in $(stat /usr/bin/* | grep -B 3 4755 | grep File: | sed -e "s/^[^\`]\+\`\([^']\+\).*/\1/"); do chmod 4755 /mnt${fn}; echo ${fn}; done
 +
</pre>
 +
I also ran this command for /bin, /sbin, /usr/bin, /usr/sbin, /usr/local/bin, /usr/local/sbin, to be sure I didn't miss anything.
 +
 +
== truncate a file ==
 +
 +
If you want to dump the contents of a file, but still keep the file then you are looking to truncate it. This is common with log files where a service may be actively writing to the log. You wish to dump the old contents, but you don't want to interrupt the service. This is common when truncating the '''utmp''' and '''wtmp''' log of of logins. The '''login''' process never creates this file, so it must already exist otherwise logging is disabled; except, in Linux the file must always exist (the '''login''' command complains, but I'm not sure how serious this is). Also, it's a binary fine, so you can easily corrupt it by inserting a byte in the wrong place. And then there may be active logins logging to the file when you want to truncate it... So don't screw it up. You can still truncate the file while it's in use. You just have to do it the correct way.
 +
 +
The following are the two most portable ways to truncate a file. The first may be slightly faster if '''printf''' is a shell built-in.
 +
<pre>
 +
printf "" > /var/log/example.log
 +
true > /var/log/example.log
 +
</pre>
 +
Less portable ways (and some simply ''wrong'' ways) to do this are commonly seen. These should be avoided:
 +
<pre>
 +
cat /dev/null > /var/log/example.log
 +
> /var/log/example.log
 +
: > /var/log/example.log
 +
truncate -s /var/log/example.log
 +
echo -n > /var/log/example.log
 +
</pre>
 +
 +
Note that this is very different than deleting and recreating the file.
 +
<pre>
 +
rm -f /var/log/example.log
 +
touch /var/log/example.log
 +
</pre>
 +
The inode will change when you recreate it. Processes that had the file open will continue to write to the file formerly associate with the old inode. From your point of view all new messages to the log file go nowhere. They certainly don't go to the log file you recreated.
 +
 +
== for loop over recursive directory tree file list ==
 +
 +
<pre>
 +
# globstar option is required for recursive directory file iteration.
 +
shopt -s globstar
 +
for ff in **/*.png; do
 +
    md5sum "$ff"
 +
done
 +
</pre>
 +
 +
== symlink every file to a directory above ==
 +
 +
Since this also symlinks directories it doesn't need to be recursive. This is useful for swapping out environments or sets of dotfiles.
 +
<pre>
 +
ln -sf ${PWD}/.* ../.
 +
ln -sf ${PWD}/* ../.
 +
</pre>
 +
 +
== USER vs. LOGNAME ==
 +
 +
The great '''USER''' versus '''LOGNAME''' question comes up from time to time. Which is which? Which one should I use?
 +
 +
'''USER''' is an environment variable defined by Bash and many other shells. '''LOGNAME''' is required by POSIX and is defined as an environment variable that must be initialized to the user's login name at the time of login, so this should be available on POSIX compliant systems even if the user's login shell is set to something other than a normal shell ('''/bin/sh''' or '''/bin/bash''') such as '''rssh''' (a Restricted Shell for OpenSSH).
 +
 +
There is an important detail to note about these environment variables versus the '''logname''' command. Both '''USER''' and '''LOGNAME''' may be modified or unset since they are ordinary environment variables, whereas the '''logname''' utility explicitly ignores '''LOGNAME''', so it will return the original login name even if the environment has been modified.
 +
 +
I favor the use of '''LOGNAME''' in my scripts.
 +
 +
== A better, more flexible shell history logger using rsyslog ==
 +
 +
This will cause shell history to be logged using the rsyslog '''logger''' command. After setting this up all commands will be logged to '''/var/log/bash-commands.log'''. This will log date, user ID, PID of the command that was run, exit status of the command, the current working directory, and the full command-line of the command. This does not log the environment the command was run with, but this includes everything else I could think of that would be useful to recreate the circumstances and context of under which a command was run. This is useful for debugging and forensics. It helps answer questions like "Why did it work before, but not now?" and "What were they doing?". Regular command history is insufficient for this. Most critically, it misses the current working directory.
 +
 
First, setup the '''rsyslog''' daemon so that it will save log messages to a separate file, '''/var/log/bash-commands.log'''. We will log using the '''local7''' facility, which does not correspond to the log severity level 7, debug, but are kept the same for the sake of preventing a mixup in the numbers.
 
First, setup the '''rsyslog''' daemon so that it will save log messages to a separate file, '''/var/log/bash-commands.log'''. We will log using the '''local7''' facility, which does not correspond to the log severity level 7, debug, but are kept the same for the sake of preventing a mixup in the numbers.
 
<pre>
 
<pre>
Line 9: Line 89:
 
</pre>
 
</pre>
  
Resart '''rsyslog''' to have the changes applied.
+
Restart '''rsyslog''' to have the changes applied.
 
<pre>
 
<pre>
 
service rsyslog restart
 
service rsyslog restart
Line 24: Line 104:
 
</pre>
 
</pre>
  
The '''PROMPT_COMMAND''' will log the epoch time as the first item in the message. This is done because '''rsyslog''' message are not always recorded in the same order they were received. A classic mistake in log file analysis is to assume that the time and order written in the log file always corresponds to the order and time the log server received the message. Having the epoch time helps clear up any ambiguity. The log could also be post processed and sorted on this field. The time stamp that '''rsyslog''' write is the time it wrote the message to the file, which may be different than the time it received the log message.
+
The '''PROMPT_COMMAND''' will log the epoch time as the first item in the message. ''This is done because '''rsyslog''' messages are not always recorded in the same order they were received''. ... Also, remember that that the '''PROMPT_COMMAND''' sends the message ''after'' the user's command has exited. I'm sort of abusing '''PROMPT_COMMAND''' here by using it as a hook. In this case it's a post-hook. A pre-hook would be better. Both would be ideal. Logging the environment would be make this even better. I might add that feature. I didn't want to make the log too difficult to read. If I add environment then turning on compression in '''rsyslog''' would be helpful because 90% of the log is likely to be the exact same environment logged over and over. Maybe a baseline environment could be logged and a diff could be logged if it ever changes from the baseline... Features for Version 2.0. This is all a bit of a hack. Maybe it makes more sense to patch features like this into the Bash binary.
 
 
Now all commands will be logged to '''/var/log/bash-commands.log'''. The logging information will include the working directory the command was run from and the exit status code.
 
  
 
== Bash Script Hints ==
 
== Bash Script Hints ==
Line 170: Line 248:
  
 
I always forget this. The main difference is how the parameters are expanded inside of quotes.
 
I always forget this. The main difference is how the parameters are expanded inside of quotes.
;"$*": expands to "$1c$2c$3c...", where '''c''' is the first character of the value of the IFS variable.
+
;"$*": expands to "$1'''c'''$2'''c'''$3'''c'''...", where '''c''' is the '''first''' character of the value of the IFS variable.
 
;"$@": expands to "$1" "$2" "$3"...
 
;"$@": expands to "$1" "$2" "$3"...
  
Line 312: Line 390:
 
This version is shorter, but I have not tested it as much as the first version.
 
This version is shorter, but I have not tested it as much as the first version.
 
<pre>
 
<pre>
function get_realpath() {
+
get_realpath() {
 
         if [ -d "$1" ]; then
 
         if [ -d "$1" ]; then
 
                 cd "$1"
 
                 cd "$1"
Line 329: Line 407:
 
         return 0
 
         return 0
 
}
 
}
 +
</pre>
 +
 +
== Move or delete files with leading dashes ==
 +
 +
The short answer is to use the '''option terminator''' option, which is simply two dashes, '''--'''.
 +
 +
If you have a file named '''-h''' and you try to rename it you will quickly see the problem here:
 +
<pre>
 +
$ mv -h.txt h.txt
 +
mv: illegal option -- h
 +
usage: mv [-f | -i | -n] [-v] source target
 +
      mv [-f | -i | -n] [-v] source ... directory
 +
</pre>
 +
The same problem happens with '''rm''' and other commands and built-ins. This really isn't Bash's fault as it just passes all arguments to the command (after it does do string escaping and filename globbing). It's up to the command to parse and handle arguments that start with '''-''' as special options, so no amount of quoting in Bash could ever eliminate the ambiguity between options and filename arguments. People often try to escape the '''-''' as a way around the problem. Again, quoting or escapes of any kind will not help. You might be aware of the obscure Bash string quoting syntax that looks like a variable expansion, but is a way to specify strings with escape sequences that are expanded by Bash before being used as arguments. It looks like this (note that hex 2d and octal 055 are the ASCII codes for the '''-''' character. Neither hex nor octal help here.):
 +
<pre>
 +
mv $'\055h.txt' h.txt
 +
mv: illegal option -- h
 +
...
 +
mv $'\x2dh.txt' h.txt
 +
mv: illegal option -- h
 +
</pre>
 +
 +
So, to specifically deal with this problem both the '''mv''' and '''rm''' commands have options to indicate when no more options follow, so they will not treat following arguments that start with a '''-''' as options. This option terminator is '''--''' (two dashes). This works with many other commands, too; although, it isn't always documented (grep, for example, uses -- as an option terminator, but the man page does not mention this feature.).
 +
<pre>
 +
mv -- -h.txt h.txt
 +
rm -- -h.txt h.txt
 +
# grep for the string, -h in, a file named -h.txt.
 +
grep -- -h -h.txt
 
</pre>
 
</pre>
  
Line 442: Line 548:
 
# ...
 
# ...
 
</pre>
 
</pre>
 
== Bashing Bash ==
 
 
When you get down to it, Bash is a piece of crap. It has no redeeming qualities except that it is ubiquitous. That is, in fact, the only reason I have put any effort into it at all. By the time I get deep into a project that involves a giant amount of Bash I usually regret it and wish I had just started the project off with Python.
 
 
That said, I seem to have learned a lot more Bash than I ever intended.
 
  
 
== Arithmetic ==
 
== Arithmetic ==
Line 498: Line 598:
 
<pre>
 
<pre>
 
FILES=$(ls -RQal | \
 
FILES=$(ls -RQal | \
      awk -v PATH=$path '{ \
+
    awk -v PATH=$path '{ \
                          if ($0 ~ /.*:$/) \
+
    if ($0 ~ /.*:$/) \
                              path = substr($0,1,length($0)-2); \
+
        path = substr($0,1,length($0)-2); \
                          else \
+
    else \
                              if ($0 ~ /^-/) \
+
        if ($0 ~ /^-/) \
                                  printf("%s%s/%s\n", PATH, path, \
+
            printf("%s%s/%s\n", PATH, path, \
                                          substr($0, match($0,"\".*\"$")+1, RLENGTH-1) \
+
            substr($0, match($0,"\".*\"$")+1, RLENGTH-1) \
                                        ) \
+
            ) \
                          }' \
+
    }' \
      )
+
    )
 
</pre>
 
</pre>
  
 
To make the quoted list of files work with a '''for''' loop you will need to set '''IFS'''.
 
To make the quoted list of files work with a '''for''' loop you will need to set '''IFS'''.
 
<pre>
 
<pre>
 +
# lost snippet of code
 +
</pre>
 +
 +
== Shell quoting strings so they are safe to pass to other commands.  ==
  
 +
Shell quote... This really belongs in the [[sed sed]] article.
 +
<pre>
 +
shellquote() {
 +
        # Single quote stdin. Escape existing single quotes.
 +
        sed -e "s/'/'\"'\"'/g" -e "s/^/'/" -e "s/$/'/"
 +
}
 
</pre>
 
</pre>
  
 
== Interview with Steve Bourne ==
 
== Interview with Steve Bourne ==
  
[http://www.computerworld.com.au/article/279011/a-z_programming_languages_bourne_shell_sh/ Interview with Steve Bourne].
+
[https://www.computerworld.com/article/3517950/the-a-z-of-programming-languages-bourne-shell-or-sh.html Interview with Steve Bourne].
  
== here file, herefile ==
+
== here file, herefile, here doc, here string, here here ==
  
 
<pre>
 
<pre>
 
#!/bin/sh
 
#!/bin/sh
process <<EOF
+
cat <<EOF
This is the
+
This is a
 
here document.
 
here document.
 
EOF
 
EOF
Line 575: Line 685:
 
</pre>
 
</pre>
  
== Assign variable the contents of a "here document" ==
+
=== Assign variable the contents of a "here document" ===
  
 
'''here doc'''
 
'''here doc'''
Line 593: Line 703:
 
</pre>
 
</pre>
  
== special forms of here docs and here strings ==
+
=== special forms of here docs and here strings ===
=== Strip leading tabs, but not spaces===
+
==== Strip leading tabs, but not spaces====
 
This form of heredoc will allow you to indent the document to be consistent with the code indentation used in the script yet still include and preserve formatting within the document itself. The only twist is that the code indentation must be TABS whereas the document indentation must be  spaces. Leading TABs will be stripped.
 
This form of heredoc will allow you to indent the document to be consistent with the code indentation used in the script yet still include and preserve formatting within the document itself. The only twist is that the code indentation must be TABS whereas the document indentation must be  spaces. Leading TABs will be stripped.
 
<pre>
 
<pre>
Line 608: Line 718:
 
echo "${DOCUMENT}"
 
echo "${DOCUMENT}"
 
</pre>
 
</pre>
== Here strings ==
+
=== Here strings ===
  
 
Here strings are like here docs but use single or double quotes instead of a string as a delimiter. Also, the leading and trailing newlines are not removed. Here strings use the '''<<<''' operator instead of '''<<''.
 
Here strings are like here docs but use single or double quotes instead of a string as a delimiter. Also, the leading and trailing newlines are not removed. Here strings use the '''<<<''' operator instead of '''<<''.
Line 703: Line 813:
 
This will delete all environment variables except for a few explicitly allowed to stay.
 
This will delete all environment variables except for a few explicitly allowed to stay.
 
<pre>
 
<pre>
unset $(env | grep -o '^[_[:alpha:]][_[:alnum:]]*' | grep -v -E '^PWD$|^USER$|^TERM$|^SSH_.*|^LC_.*')
+
unset $(env | grep -o '^[_[:alpha:]][_[:alnum:]]*' | grep -v -E '^PWD$|^LOGNAME$|^USER$|^TERM$|^SSH_.*|^LC_.*')
 
</pre>
 
</pre>
  
== Redirect entire output of a script from inside the script itself ==
+
== Redirect stderr output to log file ==
 +
 
 +
<pre>
 +
exec 2>> stderr.log
 +
</pre>
 +
 
 +
== Color stderr output a different color (color stderr red) ==
 +
 
 +
Note that this is still line-orienced (anonymous FIFO) and there are race conditions.
 +
<pre>
 +
exec 3>&2
 +
exec 2> >(sed -u 's/^\(.*\)$/'$'\e''[31m\1'$'\e''[m/' >&3)
 +
</pre>
 +
 
 +
Does essentially the same thing without sed. Uses anonymous FIFO ('''>(command)'''). Note different ANSI color code format.
 +
<pre>
 +
exec 2> >(while read line; do echo -e "\e[01;31m$line\e[0m"; done)
 +
</pre>
 +
 
 +
== Redirect output of a script from inside the script itself ==
  
 
<pre>
 
<pre>
Line 737: Line 866:
 
     TEEPID=$!
 
     TEEPID=$!
 
     # Redirect subsequent stdout and stderr output to named pipe.
 
     # Redirect subsequent stdout and stderr output to named pipe.
 +
    # Alternate forms:
 +
    #    exec >> log.log 2>&1
 +
    # or
 +
    #    exec 2>> stderr.log
 
     exec > ${PIPEFILE} 2>&1
 
     exec > ${PIPEFILE} 2>&1
 
     trap on_exit_trap_cleanup EXIT
 
     trap on_exit_trap_cleanup EXIT
Line 807: Line 940:
 
== Special Shell Variables ==
 
== Special Shell Variables ==
  
''Note the difference between '''$*''' and '''$@'''.''
+
''Note the difference between '''$*''' and '''$@'''.'' Also note that the Bash documentation seems to use the term '''argument''' and '''parameter''' synonymously, so I am redundant to make full text searches more likely to match this documentation.
  
;$*: all parameters separated by the first character of $IFS
+
;$*: all argument parameters separated by the first character of $IFS
;$@: all parameters quoted
+
;$@: all argument parameters quoted
;$#: the number of parameters
+
;$#: count of the number of argument parameters passed to command.
 
;$-: option flags set `set` or passed to shell
 
;$-: option flags set `set` or passed to shell
 
;$?: exit status of last command
 
;$?: exit status of last command
Line 817: Line 950:
 
;$$: pid of this script or shell
 
;$$: pid of this script or shell
 
;$0: name of this script of shell
 
;$0: name of this script of shell
;$_: arguments of last command (with variables expanded).
+
;$_: argument parameters of last command (with variables expanded).
  
 
== Variable Expansion and Substitution ==
 
== Variable Expansion and Substitution ==
Line 845: Line 978:
 
MY_STRING="${MY_STRING%"${MY_STRING##*[![:space:]]}"}"
 
MY_STRING="${MY_STRING%"${MY_STRING##*[![:space:]]}"}"
 
echo "|${MY_STRING}|"
 
echo "|${MY_STRING}|"
 +
</pre>
 +
 +
Here it is in function form.
 +
<pre>
 +
# This strips leading and trailing whitespace.
 +
strip ()
 +
{
 +
        local STRING="$@"
 +
        STRING="${STRING#"${STRING%%[![:space:]]*}"}"
 +
        STRING="${STRING%"${STRING##*[![:space:]]}"}"
 +
        echo -n "${STRING}"
 +
}
 
</pre>
 
</pre>
  
Line 1,066: Line 1,211:
 
</pre>
 
</pre>
  
=== read a single character key then return -- with no Enter required ===
+
=== read a single character then return with no Enter required ("press any key to continue...") ===
  
 
The following is a discussion of `stty` command. In '''Bash''' and '''Korn shell''' you can already get a single character using `read`. The following will set the variable, CHARACTER, with a single key read from stdin: `read -r -s -n 1 CHARACTER`.
 
The following is a discussion of `stty` command. In '''Bash''' and '''Korn shell''' you can already get a single character using `read`. The following will set the variable, CHARACTER, with a single key read from stdin: `read -r -s -n 1 CHARACTER`.
Line 1,076: Line 1,221:
 
<pre>
 
<pre>
 
# This reads a single character without echo.
 
# This reads a single character without echo.
 +
# "Press any key to continue..."
 
# If a variable name argument is given then it is set to a character read from stdin.
 
# If a variable name argument is given then it is set to a character read from stdin.
 
# else the variable REPLY is set to a character read from stdin.
 
# else the variable REPLY is set to a character read from stdin.
Line 1,097: Line 1,243:
 
     fi
 
     fi
 
}
 
}
 +
</pre>
 +
 +
=== The problem with '''read -t 0''' ===
 +
 +
In Bash '''read -t 0''' is fairly useless in '''stdin''' without having previously set '''stty -icanon'''.
 +
 +
<pre>
 +
# Echo all data read in a tight loop for one second.
 +
stty -echo -icanon min 0 time 0 <&0
 +
user_data=""
 +
timelimit=$((1 + $(date "+%s")))
 +
while [ $(date "+%s") -lt ${timelimit} ]; do
 +
        if read -t 0; then
 +
                read -n 1 partial
 +
                user_data=${user_data}${partial}
 +
        fi
 +
done
 +
stty echo icanon <&0
 +
echo ${user_data}
 +
 +
# Similar to the previous version, but should be more efficient.
 +
# Echo all data input over a 1 second period of time.
 +
stty -echo -icanon min 0 time 0 <&0
 +
user_data=""
 +
sleep 1
 +
while read -t 0; do
 +
      read -n 1 partial
 +
      user_data=${user_data}${partial}
 +
done
 +
stty echo icanon <&0
 +
echo ${user_data}
 
</pre>
 
</pre>
  
 
=== read with a timeout ===
 
=== read with a timeout ===
 +
 +
'''I never quite got this working again.''' I'm pretty sure it worked, but I got interrupted when I was copying my notes. Now that I've come back to this I can't get it working again. Reading online it seems that most versions of Bash 3 and Bash 4 have a bug in the built-in '''read''' command.
  
 
The Bash built-in already has a timeout option. The following solution will work under most POSIX Bourne style shells:
 
The Bash built-in already has a timeout option. The following solution will work under most POSIX Bourne style shells:
Line 1,128: Line 1,307:
 
<pre>
 
<pre>
 
mktempdir () {
 
mktempdir () {
        CLEAN_NAME=$(echo $0 | sed "s/[-_.\/]//g")
+
    CLEAN_NAME=$(echo $0 | sed -e "s/[^[:alpha:]]//g")
        NEW_TMPDIR=${TMPDIR-/tmp}/$(date "+tmp-${CLEAN_NAME}.$$.%H%M%S")
+
    NEW_TMPDIR=${TMPDIR-/tmp}/$(date "+tmp-${CLEAN_NAME}.$$.%H%M%S")
        (umask 077 && mkdir ${NEW_TMPDIR} 2>/dev/null && echo ${NEW_TMPDIR}) || return 1
+
    (umask 077 && mkdir ${NEW_TMPDIR} 2>/dev/null && echo ${NEW_TMPDIR}) || return 1
        return 0
+
    return 0
 
}
 
}
 
</pre>
 
</pre>
Line 1,138: Line 1,317:
 
<pre>
 
<pre>
 
if ! MYTEMPDIR=$(mktempdir); then
 
if ! MYTEMPDIR=$(mktempdir); then
         echo "Could not create a temporary directory."
+
         echo "Could not create a temporary directory." >&2
 
         exit 1
 
         exit 1
 
fi
 
fi
Line 1,148: Line 1,327:
 
<pre>
 
<pre>
 
if [ $(id -u) -eq 0 ]; then
 
if [ $(id -u) -eq 0 ]; then
     echo "You are root."
+
     echo "You are root. Good for you."
 
fi
 
fi
 
</pre>
 
</pre>
Line 1,154: Line 1,333:
 
<pre>
 
<pre>
 
if [ $(id -u) -ne 0 ]; then  
 
if [ $(id -u) -ne 0 ]; then  
     echo "ERROR: You must be root to run this." >&2
+
     echo "ERROR: You must be root to run this script." >&2
 +
    echo "Perhaps you forgot to use sudo." >&2
 
fi
 
fi
 
</pre>
 
</pre>
Line 1,189: Line 1,369:
 
exit 1
 
exit 1
 
</pre>
 
</pre>
 +
 +
= Bash vs. Bourne quirks =
 +
 +
== function definitions ==
 +
 +
Bash will allow function and variable definitions with dashes "-" in them. Bourne shell will not allow the following.
 +
<pre>
 +
epoch-date ()
 +
{
 +
    date "+%s"
 +
}
 +
</pre>
 +
 +
= References =
 +
 +
[http://google-styleguide.googlecode.com/svn/trunk/shell.xml Google shell style guide]

Latest revision as of 13:22, 28 April 2020


Contents

Bash, mostly

Bash function to return formatted date in days past for both GNU Linux and MacOS OSX

Pass the number of days in the past for which you want the date. If you pass nothing it will assume 0 days.

date_days_past () { days_past=${1:-0}; if ! date -v-${days_past}d +%Y%m%d 2>/dev/null; then date --date="-${days_past} day" +%Y%m%d; fi }

script intended to be sourced only

If you have a script that is intended to be sourced instead of executed directly you can add an error message in case the script is executed directly. ... You could also use the trick of exec'ing a subshell that sources the file, but that's kind of ugly.

#!/bin/echo "Source this file. Do no run it."

Replicate SUID permissions

I had a filesystem where I lost all SUID permissions on executables. I mounted the filesystem under a different server and used the following to restore the SUID bit on all executables that needed it (especially sudo). I mounted the filesystem to be repaired under /mnt. I assume it has pretty much the same layout as the reference filesystem.

for fn in $(stat /usr/bin/* | grep -B 3 4755 | grep File: | sed -e "s/^[^\`]\+\`\([^']\+\).*/\1/"); do chmod 4755 /mnt${fn}; echo ${fn}; done

I also ran this command for /bin, /sbin, /usr/bin, /usr/sbin, /usr/local/bin, /usr/local/sbin, to be sure I didn't miss anything.

truncate a file

If you want to dump the contents of a file, but still keep the file then you are looking to truncate it. This is common with log files where a service may be actively writing to the log. You wish to dump the old contents, but you don't want to interrupt the service. This is common when truncating the utmp and wtmp log of of logins. The login process never creates this file, so it must already exist otherwise logging is disabled; except, in Linux the file must always exist (the login command complains, but I'm not sure how serious this is). Also, it's a binary fine, so you can easily corrupt it by inserting a byte in the wrong place. And then there may be active logins logging to the file when you want to truncate it... So don't screw it up. You can still truncate the file while it's in use. You just have to do it the correct way.

The following are the two most portable ways to truncate a file. The first may be slightly faster if printf is a shell built-in.

printf "" > /var/log/example.log
true > /var/log/example.log

Less portable ways (and some simply wrong ways) to do this are commonly seen. These should be avoided:

cat /dev/null > /var/log/example.log
> /var/log/example.log
: > /var/log/example.log
truncate -s /var/log/example.log
echo -n > /var/log/example.log

Note that this is very different than deleting and recreating the file.

rm -f /var/log/example.log
touch /var/log/example.log

The inode will change when you recreate it. Processes that had the file open will continue to write to the file formerly associate with the old inode. From your point of view all new messages to the log file go nowhere. They certainly don't go to the log file you recreated.

for loop over recursive directory tree file list

# globstar option is required for recursive directory file iteration.
shopt -s globstar
for ff in **/*.png; do
    md5sum "$ff"
done

symlink every file to a directory above

Since this also symlinks directories it doesn't need to be recursive. This is useful for swapping out environments or sets of dotfiles.

ln -sf ${PWD}/.* ../.
ln -sf ${PWD}/* ../.

USER vs. LOGNAME

The great USER versus LOGNAME question comes up from time to time. Which is which? Which one should I use?

USER is an environment variable defined by Bash and many other shells. LOGNAME is required by POSIX and is defined as an environment variable that must be initialized to the user's login name at the time of login, so this should be available on POSIX compliant systems even if the user's login shell is set to something other than a normal shell (/bin/sh or /bin/bash) such as rssh (a Restricted Shell for OpenSSH).

There is an important detail to note about these environment variables versus the logname command. Both USER and LOGNAME may be modified or unset since they are ordinary environment variables, whereas the logname utility explicitly ignores LOGNAME, so it will return the original login name even if the environment has been modified.

I favor the use of LOGNAME in my scripts.

A better, more flexible shell history logger using rsyslog

This will cause shell history to be logged using the rsyslog logger command. After setting this up all commands will be logged to /var/log/bash-commands.log. This will log date, user ID, PID of the command that was run, exit status of the command, the current working directory, and the full command-line of the command. This does not log the environment the command was run with, but this includes everything else I could think of that would be useful to recreate the circumstances and context of under which a command was run. This is useful for debugging and forensics. It helps answer questions like "Why did it work before, but not now?" and "What were they doing?". Regular command history is insufficient for this. Most critically, it misses the current working directory.

First, setup the rsyslog daemon so that it will save log messages to a separate file, /var/log/bash-commands.log. We will log using the local7 facility, which does not correspond to the log severity level 7, debug, but are kept the same for the sake of preventing a mixup in the numbers.

echo "local7.*    /var/log/bash-commands.log" > /etc/rsyslog.d/70-bash-commands.conf

Restart rsyslog to have the changes applied.

service rsyslog restart

Add the following to the user's ~/.bashrc file.

export PROMPT_COMMAND='EXIT_STATUS=$?;logger -p local7.debug "$(date "+%s") ID=$(id -u) PID=$$ ?=$(printf "%03d" ${EXIT_STATUS}) ${PWD} $(history 1 | sed "s/^[[:space:]]*[0-9]\+[[:space:]]*[^[:space:]]\+[[:space:]]\+//")";history -a'

Source the ~/.bashrc file.

. ~/.bashrc

The PROMPT_COMMAND will log the epoch time as the first item in the message. This is done because rsyslog messages are not always recorded in the same order they were received. ... Also, remember that that the PROMPT_COMMAND sends the message after the user's command has exited. I'm sort of abusing PROMPT_COMMAND here by using it as a hook. In this case it's a post-hook. A pre-hook would be better. Both would be ideal. Logging the environment would be make this even better. I might add that feature. I didn't want to make the log too difficult to read. If I add environment then turning on compression in rsyslog would be helpful because 90% of the log is likely to be the exact same environment logged over and over. Maybe a baseline environment could be logged and a diff could be logged if it ever changes from the baseline... Features for Version 2.0. This is all a bit of a hack. Maybe it makes more sense to patch features like this into the Bash binary.

Bash Script Hints

Originally from [1]. Even more interesting Bash info from Anthony Thyssen.

File:script.hints

Add timestamps to each line of output

awk '{now=strftime("%s "); print now $0}'

The first example adds the epoch timestamp. The second example adds RFC-3339 timestamps.

iostat -z -N 2 | awk '{now=strftime("%s "); print now $0}'
iostat -z -N 3 | awk '{now=strftime("%F %T%z "); print now $0}'

test if running in interactive terminal or in a pipeline

This checks if file descriptor 0 is a tty or not. If not a tty then it is running in a pipeline, so return 1.

is_pipeline() {
    [[ -t 0 ]] && return 1 || return 0 
}

# Ubuntu systems check $PS1.
## if not running interactively, don't do anything
# [ -z "$PS1" ] && return

fork bomb

This is funny because : is a valid function name in Bash. Don't forget the space after the first curly brace, { . If you run this as root on a machine you will likely need to reboot it.

:(){ :|:&};:

To understand what it does you can replace : with something easier to read, such as, fork_bomb:

fork_bomb(){ fork_bomb|fork_bomb& };fork_bomb

echo variable with newlines in it

Always quote variables if you want echo to preserve newlines. This is one of those simple things that I know, yet always forget when I'm trying to remember why echo hates newlines. For example say you want to inject a few default SSH keys into an authorized_keys file in a rootfs you have mounted under ${ROOTFS_IMAGE}:

AUTHORIZED_KEYS="$(cat /root/.ssh/*.pub)
ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAgEAvMZJJ2XqN0IYcH4QJrzs+d3oULswRV/k2P25WMixRN+SXSU4Sh9QlO6YO6hixeksAMAtjy+41+c4javtSYkPMITEpbGHElUiHTqltFmdIsnKjQAA+a/wr6MR5iQCCySzYZIcAWaU8Qvs9KIdYEvv9gI7M33wfd6jxYPtsatnZHR+6gcokMQc/urleXzgr8ocoULN46t0+MKlEaNv4mAawf5mf531PZIS3h0+zNsHth2pwqKn0BsY7awvxRcEOToFfqYbMXmnufYvjUpRtf5Ehhhv5MvdgBYb1/POKvqstH/MiNS1labKdvm1JHzdTD6OnQQaSdzVYSZYOGXD2ejWRa8fselvK4OLtDnqMPo5Gknrn4WOhQohW2dDYkt2eKAw9jv3bkmPmlyUKwrnZ8bVdEkGmOnRmHxLdCXrm4Cut6OEoCyQfGTfiNa7tznSeLx+Vklobr/XvWhOv4gX6+j8OXCpOv0F/C5OZGf/yQPGMty70PcqigM8dY+vTct5auTpC3M6UBTmrdu42VtPTgM8no+95Lz62klKIHKGtBYCXt4eUIok9C30p5pRAu93gU7ZtEJEt/hQut3vJgauvfSQve9Q+vMAvGYJ/HfbNQECJ1/dSMeNLNx16aJv25K/INYLLb2lO3y44suPIvZCWAvnaPrEEMRrZ0ENfbXAQislqN0= root@gateway.example.com
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABDadBAQC4SbAeI58uf1UjNGjGR8CCvQibkhPvMrv/2hGfiV+yM+pVLdFNBoAVXDdxBE6giNlF9/g6uCxwGypq6Lgk02VcleaNLKNIYRfqGjmAV2ZPP/Yi+qZg44jpfTb+41i2D2gYQncfWz/wFC2RNK8+6fMDejK2RYClpfVWvCooE9oUQjlpG/rYr194GnIwoX4P6tc4FsIIdttLJtBF/npbZJkbpaiox145vSQHDrT/PykoRMX1WOCC3JihqYY0LRylG+JMWB1ZBoORtI7Q9y2xn+QY5v2DO4uKTWocSL8MHiTzR4PK62+UDNwOTuD93Fv2DMFzfgCxtUyC0nns/DwJUCvB noah
ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuynJuudXzQSadqJPq1fz3PZK8it/ayHErgHfcMq/bCFn6rShqhSoKgwdfoa99es/lQ+huCUPy5SUSrcA1mpJ4rBuOWafSrEOXYKsINRk5hnUiSNV/2F7x+VBU4weh10Zkhgpaqmct6wkPVio+E15Hk+z9IGQrT9QxisOv31hPYNZ9wnUzxcHgJ6SfAZdIPm3FGQxf8sJKlEaNC061k7zH6InwTppqBas3djUqBReOsWP+3/Ccq6XL/OsxGVAcH/4HOL6umhcb1VDJsWlrI5HuB9FgKcsG3BPVhPDCh89WEjb03c/5dG98IYjNUQDoMsUtfQqbQsbCNRWfmsx+LFDTw== jenkins@buildbot.example.com
"
# Forget the quotes and you'll mess up the authorized_keys file.
echo "${AUTHORIZED_KEYS}" >> ${ROOTFS_IMAGE}/root/.ssh/authorized_keys

test if variable is not set versus empty or null

It can be tricky to determine if a shell variable is unset versus merely empty.

# This is ambiguous.
if [ -z "${FOO}" ]; then echo "empty or unset"; fi

One trick is to use Bash default expansion on the variable in question with two different default values. If the variable, FOO, is unset then the left side will expand to "X" while the right side will expand to "Y", so the condition will be false. If FOO is set, even if empty, then the expansion will not use the default values of "X" and "Y", so the left and right sides will be the same and the condition will be true.

if [ "${FOO-X}" == "${FOO-Y}" ]; then echo "set"; else echo "unset"; fi

The following function will encapsulate this for an unambiguous test. This is for Bash only because it uses the Bash indirect variable syntax (the !1 part). I have not figured out how to get both variable expansion and indirection to work at the same time in a Bourne shell script.

function defined { [ "${!1-X}" == "${!1-Y}" ]; }

This shows how the defined function is used:

FOO=
if defined FOO; then echo "set"; else echo "unset"; fi
unset FOO
if defined FOO; then echo "set"; else echo "unset"; fi
if ! defined FOO; then echo "unset"; else echo "set"; fi

alternate screen and normal screen restore

#!/bin/bash
# turn off cursor
tput civis 2>/dev/null
# switch to alternate screen
tput smcup 2>/dev/null
# trap to restore normal screen and cursor on exit
trap "exitcode=$?; tput rmcup; tput cnorm 2>/dev/null; exit ${exitcode}" SIGINT EXIT

dynamic COLUMNS and LINES with SIGWINCH in Bash

Bash automatically defines COLUMNS and LINES in interactive shells. These variables are not set in scripts. So if you want to use COLUMNS and LINES in a script you must do two things -- you must explicitly set COLUMNS and LINES, and you must update these values dynamically in response to changes in the terminal window size (that is, you must handle the SIGWINCH signal).

function winch_handler() {
        # This adds post-processing after the terminal handles SIGWINCH.
        # First, pass the SIGWINCH back to the terminal because
        # we can't get the new size until the terminal sees SIGWINCH.
        trap - SIGWINCH
        kill -SIGWINCH $$
        # Now tput can query the terminal for the new size.
        COLUMNS=$(tput cols)
        LINES=$(tput lines)
        # Restore this winch handler so it will respond to future WINCH signals.
        trap "winch_handler" SIGWINCH
}
# Call the winch_handler to both initialize COLUMNS and LINES, and
# install the winch_handler trap.
winch_handler

# This demonstrates how it works.
LAST_COLUMNS=$(tput cols)
LAST_LINES=$(tput lines)
while :; do
        if [ ${LAST_LINES} -ne ${LINES} -o ${LAST_COLUMNS} -ne ${COLUMNS} ]; then
                echo "New size: ${COLUMNS}, ${LINES}"
                LAST_COLUMNS=$(tput cols)
                LAST_LINES=$(tput lines)
        fi
        sleep .1;
done

random numbers

Sometimes I need to randomly choose a network bridge interface. I have four bridges, so this means I need a random number in the range 0 through 3.

bridge_name="br$[ ( $RANDOM % 4 ) ]"

This gives a random number in the range 1 through 6

echo $[ ( $RANDOM % 6) + 1 ]

Bash $* vs $@

I always forget this. The main difference is how the parameters are expanded inside of quotes.

"$*"
expands to "$1c$2c$3c...", where c is the first character of the value of the IFS variable.
"$@"
expands to "$1" "$2" "$3"...

Generally I find "$@" to be most useful when looping over arguments that are filenames. This should work correctly even if the filenames have spaces in them.

for filename in "$@"; do
        echo ${filename}
done

pairwise operations

This script will run a command on a list of arguments grouped in pairs. It does not run the command on every possible combination of pairs of arguments. It just runs the command on the pairs in order. For example:

pairwise echo 1 2 3 4 5
1 2
2 3
3 4
4 5

You can also pass shell glob patterns. For example, the following will echo pairs of file names in directory order.

pairwise echo *        
frame00000.png frame00001.png
frame00001.png frame00002.png
frame00002.png frame00003.png
frame00003.png frame00004.png
frame00004.png frame00005.png

pairwise

#!/bin/sh

COMMAND=$1
shift

fn1=''
for fn in "$@"; do
	if [ -z "${fn1}" ]; then
		fn1=${fn}
		continue
	fi
	"${COMMAND}" "${fn1}" "${fn}"
	fn1=${fn}
done

delta time between two epochs

This formats the difference between two times as "N hours, N minutes, N seconds". The times should be passed as epoch time (date "+%s" "$@").

# This prints the hours,minutes,seconds between two epoch times.
delta_epoch ()
{
        time_start=$1
        time_end=$2
        delta_time=$((time_end - time_start))
        if [ ${delta_time} -lt 0 ]; then
                delta_time=$((-1 * delta_time))
        fi
        delta_s=$((delta_time % 60))
        delta_m=$(((delta_time / 60) % 60))
        delta_h=$((delta_time / 3600))
        echo "${delta_h} hours, ${delta_m} minutes, ${delta_s} seconds"
}

Canonical real path to a file in pure POSIX

There is frequently the need to determine the real path to a file (the full canonical path). The Linux readlink command has the ability to do this, but this is a non-POSIX feature and is not available on BSD platforms such as OS X.

The trivial solution to this is to do the following. This has limitations, but it is simple. Unfortunately, I forget the limitations at the moment and I have run out of time to edit this. Better solutions are shown after this one.

realpath=$(cd "$(dirname "$0")"; pwd -P)

The following works similar to the `readlink` command. It's amazing how complicated this gets. I have a slightly simpler version shown after this one. BUGS: On Linux executing `cd //` will put you in the root directory, but `pwd` will report // as the directory. Oddly, if you execute `cd ///` or `cd ////` or more then `pwd` will report /. On BSD this works as expected; that is, after executing `cd //` then `pwd` will report / as expected.

#!/bin/sh
canonical () {
        # This returns the canonical path to the argument.
        # The argument must exist as either a dir or file.
        # This works in a pure POSIX environment.
        
        if [ -d "${1}" ]; then
                # `cd` requires execute permission on the dir.
                if [ ! -x "${1}" ]; then
                        canon=""
                        return 1
                fi      
                oldwd=$(pwd)
                cd ${1}
                canon=$(pwd)
                # Check special case of `pwd` on root directory.
                if [ -n "${canon#/*}" ]; then
                        canon=${canon}/
                fi
                cd "${oldwd}"
        else
                # At this point we know it isn't a dir.
                # But if it looks like a dir then error.
                if [ -z "${1##*/}" ]; then
                        canon=""
                        return 1
                fi
                # It looks like a path to a file.
                # Test the if the path before the file is a dir.
                dirname=$(dirname "$1")
                if [ -d "${dirname}" ]; then
                        # `cd` requires execute permission on the dir.
                        if [ ! -x "${1}" ]; then
                                canon=""
                                return 1
                        fi
                        oldwd=$(pwd)
                        cd "${dirname}"
                        canon=$(pwd)
                        # Check special case of `pwd` on root directory.
                        if [ -z "${canon#/*}" ]; then
                                canon=/$(basename "$1")
                        else
                                canon=${canon}/$(basename "$1")
                        fi
                        cd "${oldwd}"
                else
                        # It isn't anything so error.
                        canon=""
                        return 1
                fi
        fi
        echo "${canon}"
        return 0
}
canonical "$1"
exit $?

alternate version of canonical realpath

This version is shorter, but I have not tested it as much as the first version.

get_realpath() {
        if [ -d "$1" ]; then
                cd "$1"
        else
                if [ "${1}" != "${1%/*}" ]; then
                        cd "${1%/*}"
                fi
        fi
        realpath="$(pwd -P)"
        cd - >/dev/null
        if [ -d "$1" ]; then
                echo "${realpath}"
        else
                echo "${realpath}/${1##*/}"
        fi
        return 0
}

Move or delete files with leading dashes

The short answer is to use the option terminator option, which is simply two dashes, --.

If you have a file named -h and you try to rename it you will quickly see the problem here:

$ mv -h.txt h.txt
mv: illegal option -- h
usage: mv [-f | -i | -n] [-v] source target
       mv [-f | -i | -n] [-v] source ... directory

The same problem happens with rm and other commands and built-ins. This really isn't Bash's fault as it just passes all arguments to the command (after it does do string escaping and filename globbing). It's up to the command to parse and handle arguments that start with - as special options, so no amount of quoting in Bash could ever eliminate the ambiguity between options and filename arguments. People often try to escape the - as a way around the problem. Again, quoting or escapes of any kind will not help. You might be aware of the obscure Bash string quoting syntax that looks like a variable expansion, but is a way to specify strings with escape sequences that are expanded by Bash before being used as arguments. It looks like this (note that hex 2d and octal 055 are the ASCII codes for the - character. Neither hex nor octal help here.):

mv $'\055h.txt' h.txt
mv: illegal option -- h
...
mv $'\x2dh.txt' h.txt
mv: illegal option -- h

So, to specifically deal with this problem both the mv and rm commands have options to indicate when no more options follow, so they will not treat following arguments that start with a - as options. This option terminator is -- (two dashes). This works with many other commands, too; although, it isn't always documented (grep, for example, uses -- as an option terminator, but the man page does not mention this feature.).

mv -- -h.txt h.txt
rm -- -h.txt h.txt
# grep for the string, -h in, a file named -h.txt.
grep -- -h -h.txt

Enter ASCII control codes and unprintable characters

If you have a filename with weird ASCII characters or unprintable characters then you may have trouble specifying the filename on the command-line. It can be difficult to even see which weird characters are in the filename when you run `ls`. The filename may have unsupported unicode or control codes embedded to deliberately make it difficult to delete or find. If the filename looks like it has command-line options embedded in it then see Removing_files_with_weird_names.

This creates an empty file with a filename that contains an ASCII Escape control code.

touch $'\033'.foobar
<pre>
When you run `ls *.foobar` you will see one of the following depending on your environment setting for '''LS_OPTIONS''':
<pre>
?.foobar
\033.foobar

The second form will be shown if the --escape option is added to the `ls` command or to your LS_OPTIONS environment variable. The --escape option causes `ls` to to print the octal ASCII escape code for unprintable characters.

Notice how the ESC control code was specified in the `touch` command. The string, $'\033' , is a form of Bash variable expansion for constants. This is one way Bash allows you to enter non-printing characters.

Get the really real 'real user ID'

If you run a script inside of `sudo` then the real and effective users are both 'root'. Using `id -r` doesn't work. The following will give the real user name and real uid of the user that owns the current terminal running the script. That's usually what you want.

REAL_USERNAME=$(stat -c '%U' $(tty))
REAL_UID=$(stat -c '%u' $(tty))

or

REAL_USERNAME=$(stat -c '%U' $(readlink /proc/self/fd/0))
REAL_UID=$(stat -c '%u' $(readlink /proc/self/fd/0))

Making a script safe to be run from a daemon

This will ensure that a script is totally disconnected from input and output. If a daemon runs a script in the background and that script generates output then the daemon may block when waiting for the script to finish. The script child process will show up as <defunct>. The reason is because the kernel thinks the child is dead, but will not cleanup the pid information until all child output has been flushed. In this case if the child prints anything to stdout or stderr and the parent does not read that data then parent may block on waitpid. This is usually not a problem when a script is run from a foreground process because the parent process is connected to a TTY. The TTY will automatically read stdout and stderr, so any output from the parent or child gets flushed.

This is similar to a common problem with Open SSH where a client will not exit even after you exit from the remote server. See "Why does ssh hang on exit?" in the OpenSSH FAQ (see also 3.10).

It can be tricky to be sure that a script never generates any output, but you can get around this problem by closing the standard file descriptors.

# Close stdin, stdout, and stderr to be sure this script 
# cannot generate output and cannot accept input.
exec 0>&- 1>&- 2>&-

Daemonize a bash script by double backgrounding it

There are a few ways to do this. I use the double background convention.

(./daemon_script.sh) & ) &

or

(./daemon_script.sh > /dev/null 2>/dev/null & ) &

mutex

See also #Locking with flock to prevent multiple instances running.

This type of mutex is useful in preventing cron jobs from running again if the previous cron has not finished. This prevents parallel runs of a script that uses this lock convention. If you need a simple barrier to prevent a script from running more than once then start with this function. There are other more robust ways to do this, but most require external dependencies. If you don't mind some external dependencies then you can look at kernel.org's flock (Linux only) or run-one (Linux only).

As far as I can tell, there is no best way to do this.

This is simple and short. It suffers from the stale lock problem, but I don't consider that a problem because at least it fails gracefully with an error. Adding stale lock support adds a lot of complexity. At that point it's better to use something more complex and better tested.

# One problem with this is that the trap might overwrite a trap in a different part of the script.
mutex_lock() {
    LOCK_ROOT="/var/lock"
    LOCK_NAME="$(basename $0)"
    LOCK_DIR="${LOCK_ROOT}/${LOCK_NAME}"
    LOCK_FILENAME="${LOCK_DIR}/${LOCK_NAME}.pid"
    if mkdir "${LOCK_DIR}"; then
        trap 'rm -f "${LOCK_FILENAME}"; rmdir "${LOCK_DIR}" 2>/dev/null' EXIT INT
        echo "$$" >> "${LOCK_FILENAME}"
        return 0
    else
        echo "ERROR: cannot create lock." >&2
        echo "PID of existing lock: $(cat "${LOCK_FILENAME}")" >&2
        return 1
    fi
}

Example usage:

if ! mutex_lock; then
        exit 1
fi

Locking with flock to prevent multiple instances running

See also #mutex

This is a common idiom that you see on systems where the `flock` utility is available (most Linux systems). The `flock` utility is a command-line interface to the flock system call.

Simple add the following near the top of your script. What this does is check to see if this script was run inside of a `flock` command. If FLOCK_SET is set then it means the calling `flock` succeeded so we just continue on with the script. If FLOCK_SET is not set then that means we have to re-run the script inside of a `flock` command which will set FLOCK_SET. Note the use of `exec`. This replaces the current process, so if `flock` fails it will exit entire script. It is easy to overlook this and think that there is a bug in the logic. You might otherwise think that if `flock` fails then the rest of the script would continue after the if expression. Have no fear, if the `flock` fails then the entire script will fail to run.

# Set lockfilename to suite your application.
# lockfilename="/tmp/$0.lock"
lockfilename="/var/lock/$0"
if [ -z "$FLOCK_SET" ] ; then
    exec env FLOCK_SET=1 flock -n "${lockfilename}" "$0" "$@"
fi

# The rest of your script runs here.
# ...

Arithmetic

Bash can't do floating point -- not even division and multiplication. You can pipe through `bc` or `dc`.

In this `bc` example the "scale=4" part of the expression sets the output display precision to 4 decimal places.

echo "scale=4; 1/2" | bc
.5000

If you need trig functions then you need to add the --mathlib option. The 's()' function is sine in `bc` syntax.

 echo "scale=4; s(1/2)" | bc --mathlib
.4794

The `dc` calculator is RPN. The "4 k" part of the following expression sets precision to 4 decimal places.

$ echo "4 k 1 2 / p" | dc
.5000

I find a lot of systems that do not have `bc` or `dc` installed, so I often use `awk`. For example, say some command output numbers in two fields, bytes and seconds. To get bytes per second you could use `awk`. In this example I just use `echo` to pipe sample data to `awk`:

echo "104857600 0.767972" | awk '{print $1 / $2}'

Using 1048576 bytes in a MegaByte:

echo "104857600 0.767972" | awk '{printf ("%4.2f MB/s\n", $1 / $2 / (1024*1024))}'
130.21 MB/s

Using 1000000 bytes in a MegaByte as `dd` does:

echo "104857600 0.767972" | awk '{printf ("%4.2f MB/s\n", $1 / $2 / (1000*1000))}'
136.54 MB/s

Get a list of files for iteration

This is short and simple, plus it will ignore .svn directories.

FILES=$(find . -path "*/.svn" -prune -o -print)

This will quote the filenames. It does not use `find` and does not filter out .svn directories:

FILES=$(ls -RQal | \
    awk -v PATH=$path '{ \
    if ($0 ~ /.*:$/) \
        path = substr($0,1,length($0)-2); \
    else \
        if ($0 ~ /^-/) \
            printf("%s%s/%s\n", PATH, path, \
            substr($0, match($0,"\".*\"$")+1, RLENGTH-1) \
            ) \
    }' \
    )

To make the quoted list of files work with a for loop you will need to set IFS.

# lost snippet of code

Shell quoting strings so they are safe to pass to other commands.

Shell quote... This really belongs in the sed sed article.

shellquote() {
        # Single quote stdin. Escape existing single quotes.
        sed -e "s/'/'\"'\"'/g" -e "s/^/'/" -e "s/$/'/"
}

Interview with Steve Bourne

Interview with Steve Bourne.

here file, herefile, here doc, here string, here here

#!/bin/sh
cat <<EOF
This is a
here document.
EOF

By default shell variable expansion is done, so you must escape characters like $ and `. If "EOF" is quoted then no shell variable expansion is done. Characters like $ and ` are safe. This may also be written as \EOF. If -EOF starts with a dash then leading whitespace in the here document is stripped. This is simply to allow the here document to be indented and not have the indentation appear as part of the here document.

#!/bin/sh
FILENAME=${1}
gnuplot <<PLOT_EOF
set terminal x11 persist
set title "${FILENAME}"
plot "${FILENAME}" using 1:2 with linespoints
PLOT_EOF

Cat into /etc/network/interfaces from a herefile.

address_eth0="10.10.10.100"
address_eth1="10.10.10.101"
netmask="255.255.255.0"
gateway="10.10.10.1"
dns_nameservers="${gateway}"
dns_search="example.net"
cat > /etc/network/interfaces <<EOF_INTERFACES
auto lo
iface lo inet loopback

auto br0
iface br0 inet static
    address ${address_eth0}
    netmask ${netmask}
    gateway ${gateway}
    bridge_ports eth0
    bridge_hello 1
    dns-nameservers ${dns_nameservers}
    dns-search example.net

auto br1
iface br1 inet static
    address ${address_eth1}
    netmask ${netmask}
    gateway ${gateway}
    bridge_ports eth1
    bridge_hello 1
    dns-nameservers ${dns_nameservers}
    dns-search example.net
EOF_INTERFACES

Assign variable the contents of a "here document"

here doc here file

Here documents or here files can be assigned to a variable.

#!/bin/sh
DOCUMENT=$(cat <<END_HEREDOC
This is the documentation for this script.
The closing parenthesis must come after the delimiter, 'END_HEREDOC',
and it must be on its own line.
END_HEREDOC
)
# Note the quotes. Echo will remove formatting if DOCUMENT is not quoted.
echo "${DOCUMENT}"

special forms of here docs and here strings

Strip leading tabs, but not spaces

This form of heredoc will allow you to indent the document to be consistent with the code indentation used in the script yet still include and preserve formatting within the document itself. The only twist is that the code indentation must be TABS whereas the document indentation must be spaces. Leading TABs will be stripped.

#!/bin/sh
DOCUMENT=$(cat <<-END_HEREDOC
        This is the documentation for this script. Note that the leading
        indentation should be made with TABS, not SPACES.
        The closing parenthesis must come after the delimiter, 'END_HEREDOC',
        and must be on its own line.
END_HEREDOC
)
# Note the quotes. Echo will remove formatting if DOCUMENT is not quoted.
echo "${DOCUMENT}"

Here strings

Here strings are like here docs but use single or double quotes instead of a string as a delimiter. Also, the leading and trailing newlines are not removed. Here strings use the <<<' operator instead of <<.

cat <<< "Hello world."
cat <<< "
Hello
world."
cat <<< "
Hello
world.
"

Check if a web page exists or not

There must be a better way than this. I was surprised that curl doesn't offer an option to detect HTTP response.

curl --head --silent --no-buffer http://www.example.org/foobar.html | grep -iq "200 OK"

Example use in an 'if' statement:

if curl --head --silent --no-buffer http://www.example.org/foobar.html | grep -iq "200 OK"; then 
    echo "Web page exists"
fi

Example use handling additional HTTP response codes:

case "$(curl --head --silent --no-buffer http://www.example.org/foobar.html)" in
    *"200 OK"*) echo "200 HTTP response";;
    *"404 Not Found"*)  echo "404 HTTP response";;
    *)  exit_code=$?
        if [ ${exit_code} -ne 0 ]; then 
            echo "ERROR: curl failed with exit code ${exit_code}"
        else 
            echo "Unhandled HTTP response"
        fi
    ;;
esac

Draw a circle in ASCII

This uses `awk` for the math. The sequence in incremented by a fraction, 0.4, each time so that there is some overlap to make the circle smoother. You could use `seq 1 57`, but there will be gaps and rough edges.

tput clear;(seq 1 .4 57|awk '{x=int(11+10*cos($1/9));y=int(22+20*sin($1/9));system("tput cup "x" "y";echo X")}');tput cup 22 0

Some systems don't have the `seq` command. The following will work on a greater variety of platforms:

tput clear;(yes|head -n 114|cat -n|awk '{x=int(11+10*cos($1/18));y=int(22+20*sin($1/18));system("tput cup "x" "y";echo X")}');tput cup 22 0

That will render this fine quality circle (fits on an 80x24 console):


             XXXXXXXXXXXXXXXXXX
          XXXX                 XXX
        XX                        XXX
      XX                            XX
    XX                                X
   XX                                  XX
  XX                                    X
  X                                      X
  X                                      X
  X                                      X
  X                                      X
  X                                      X
  X                                      X
  XX                                    X
   XX                                  XX
    XX                                XX
      XX                            XX
        XX                        XX
          XXXX                 XXX
             XXXXXXXXXXXXXXXXXX

There should be a way to do it without `awk`... Maybe `join` or `paste` would help in this case. Here is a start:

seq 1 56 | sed -e 's/\(.*\)/c(\1 \/ 9)/' | bc

Then it just starts to get silly:

tput clear;(seq 1 0.4 57|awk '{x=int(11+10*cos($1/9));y=int(22+20*sin($1/9));system("tput cup "x" "y";echo X")}');tput cup 8 15;echo X;tput cup 8 28;echo X;(seq 16 0.4 21.6|awk '{x=int(11+6*cos($1/3));y=int(22+12*sin($1/3));system("tput cup "x" "y";echo X")}');tput cup 22 0

Clear all environment variables

This will delete all environment variables except for a few explicitly allowed to stay.

unset $(env | grep -o '^[_[:alpha:]][_[:alnum:]]*' | grep -v -E '^PWD$|^LOGNAME$|^USER$|^TERM$|^SSH_.*|^LC_.*')

Redirect stderr output to log file

exec 2>> stderr.log

Color stderr output a different color (color stderr red)

Note that this is still line-orienced (anonymous FIFO) and there are race conditions.

exec 3>&2
exec 2> >(sed -u 's/^\(.*\)$/'$'\e''[31m\1'$'\e''[m/' >&3)

Does essentially the same thing without sed. Uses anonymous FIFO (>(command)). Note different ANSI color code format.

exec 2> >(while read line; do echo -e "\e[01;31m$line\e[0m"; done)

Redirect output of a script from inside the script itself

#!/bin/sh

# This demonstrates printing and logging output at the same time.
# This works by starting `tee` in the background with its stdin
# coming from a named pipe that we make; then we redirect our
# stdout and stderr to the named pipe. All pipe cleanup is handled
# in a trap at exit.

# This is the exit trap handler for the 'tee' logger.
on_exit_trap_cleanup ()
{
    # Close stdin and stdout which closes our end of the pipe
    # and tells `tee` we are done.
    exec 1>&- 2>&-
    # Wait for `tee` process to finish. If we exited here then the `tee`
    # process might get killed before it hand finished flushing its buffers
    # to the logfile.
    wait $TEEPID
    rm ${PIPEFILE}
}
tee_log_output ()
{
    LOGFILE=$1
    PIPEFILE=$(mktemp -u $(basename $0)-pid$$-pipe-XXX)
    mkfifo ${PIPEFILE}
    tee ${LOGFILE} < ${PIPEFILE} &
    TEEPID=$!
    # Redirect subsequent stdout and stderr output to named pipe.
    # Alternate forms:
    #     exec >> log.log 2>&1
    # or
    #     exec 2>> stderr.log
    exec > ${PIPEFILE} 2>&1
    trap on_exit_trap_cleanup EXIT
}

LOGFILE="$0-$$.log"
echo "Logging stdin and stderr output to logfile: ${LOGFILE}"
tee_log_output ${LOGFILE}
date --rfc-3339=seconds
echo "command: $0"
echo "pid:     $$"
sleep 2
date --rfc-3339=seconds

This works only in Bash 4.x.

#!/bin/sh
# This will send output to a log file and to the screen using an
# unamed pipe to `tee`. This works only in Bash 4.x.
exec > >(tee -a ${LOGFILE})

date --rfc-3339=seconds
echo "command: $0"
echo "pid:     $$"
sleep 1
date --rfc-3339=seconds

Turn off bash history for a session

set +o history

Rename a group of files by extension

For example, rename all images from foo.jpg to foo_2.jpg.

This is somewhat more clear:

for filename in *.jpg ; do mv $filename `basename $filename .jpg`_2.jpg; done

This is more "correct" and doesn't require `basename`:

for filename in *.jpg ; do mv $filename ${filename%.jpg}_2.jpg; done

Usage Function

exit_with_usage() {
    local EXIT_CODE="${1:-0}"

    if [ ${EXIT_CODE} -eq 1 ]; then
        exec 1>&2
    fi

    echo "TODO: This script does something useful."
    echo "Usage: $0 [-h | --help]"
    echo "  -h --help         : Shows this help."

    exit "${EXIT_CODE}"
}

Special Shell Variables

Note the difference between $* and $@. Also note that the Bash documentation seems to use the term argument and parameter synonymously, so I am redundant to make full text searches more likely to match this documentation.

$*
all argument parameters separated by the first character of $IFS
$@
all argument parameters quoted
$#
count of the number of argument parameters passed to command.
$-
option flags set `set` or passed to shell
$?
exit status of last command
$!
pid of last background command
$$
pid of this script or shell
$0
name of this script of shell
$_
argument parameters of last command (with variables expanded).

Variable Expansion and Substitution

Bash can do some freaky things with variables. It can do lots of other substitutions. See "Parameter Expansion" in the Bash man page.

  • ${foo#pattern} - deletes the shortest possible match from the left
  • ${foo##pattern} - deletes the longest possible match from the left
  • ${foo%pattern} - deletes the shortest possible match from the right
  • ${foo%%pattern} - deletes the longest possible match from the right
  • ${foo:=text} - Use and assign default value. If $foo exists and is not null then return $foo. If $foo doesn't exist then create it; set value to 'text'; and return 'text'.
  • ${foo:-text} - Use default value. If $foo exists and is not null then return $foo, else return 'text'. This does not create $foo.
  • ${foo/pattern/replacement} - replace first instance of pattern with replacement. replacement may be empty.
  • ${foo//pattern/replacement} - replace all instances of pattern with replacement. replacement may be empty.

variable expansion to manipulate filenames and paths

strip off white space

Note that this is a little more complicated than one might first imagine. First we strip the leading, then we strip the trailing. Each strip requires two variable expansions.

MY_STRING="   Hello World!   "
echo "|${MY_STRING}|"
# strip leading
MY_STRING="${MY_STRING#"${MY_STRING%%[![:space:]]*}"}"
echo "|${MY_STRING}|"
# strip trailing
MY_STRING="${MY_STRING%"${MY_STRING##*[![:space:]]}"}"
echo "|${MY_STRING}|"

Here it is in function form.

# This strips leading and trailing whitespace.
strip ()
{
        local STRING="$@"
        STRING="${STRING#"${STRING%%[![:space:]]*}"}"
        STRING="${STRING%"${STRING##*[![:space:]]}"}"
        echo -n "${STRING}"
}

strip off any one extension on a file name (not greedy)

MY_FILENAME="my_video.project.copy.mov"
echo "${MY_FILENAME%.*}"
my_video.project.copy

strip off all extensions on a file name (greedy)

MY_FILENAME="my_video.project.copy.mov"
echo "${MY_FILENAME%%.*}"
my_video

strip off the .tar.gz extension on a file name.

MY_TARBALL="openssl-1.0.0d.tar.gz"
echo "${MY_TARBALL%.tar.gz}"
openssl-1.0.0d

strip off trailing slash if there is one in paths

MY_PATH="/var/log/apache2/"
MY_PATH="${MY_PATH%/}"
echo "${MY_PATH}"
/var/log/apache2

Note that stripping it more than once is harmless:

MY_PATH="/var/log/apache2/"
MY_PATH="${MY_PATH%/}"
MY_PATH="${MY_PATH%/}"
MY_PATH="${MY_PATH%/}"
echo "${MY_PATH}"
/var/log/apache2

strip the last directory in the path

MY_PATH="/var/log/apache2/"
MY_PATH="${MY_PATH%/*}"
echo "${MY_PATH}"
/var/log

get the last path element

You have to first strip the trailing slash.

MY_PATH="/var/log/apache2/"
MY_PATH="${MY_PATH%/}"
MY_PATH="${MY_PATH##*/}"
echo "${MY_PATH}"
apache2

strip off leading slash if there is one

MY_PATH="/var/log/apache2/"
MY_PATH="${MY_PATH#/}"
echo "${MY_PATH}"
var/log/apache2/

brace expansion versus backtick expansion for command substitution

Backtick expansion works in even the oldest Bourne shell variant. It cannot be nested without quoting.

echo `ls /boot/`

Brace expansion works in any POSIX Bourne shell (sh, ash, dash, bash, etc...).

 
echo $(ls /boot/*$(uname -r)*)

Although you can nest backticks if you quote the inner backticks:

echo `ls /boot/*\`uname -r\`*`

quote output in echo to preserve newlines

Echo converts newlines to spaces. This can be useful for substituting in loops. Quoting the argument will preserve the newlines.

This converts newlines to spaces:

echo $(ls /boot/)

The following preserves the newlines output from `ls`:

echo "$(ls /boot/)"

absolute and relative paths

Convert a relative path to a absolute path. It is stupid that there is not a command to do this. This does not effect the current working directory. This finds the absolute full path to $1:

echo "absolute path: `cd $1; pwd`"

Get the absolute path of the currently running script.

abs_path_here=`echo -n \`pwd\` ;( [ \`dirname \$0\` == '.' ] && echo ) || echo “/\`dirname \$0\`”`

Statements

Loop on filenames in a directory

for foo in *; do {
  echo ${foo}
}; done

Loop on lines in a file

for foo in $(cat data_file.txt); do {
  echo ${foo}
}; done

while loop

This is kind of like `watch`:

while sleep 1; do lsof|grep -i Maildir; done

read -- get input from user

In Bash, the builtin command, `read`, is used to get input from a user. It will read input into a variable named REPLY by default or into a given variable name.

read
echo $REPLY
read YN
echo $YN

Remember, `read` is a builtin command, so to get information on using it use `help read`, not `man read`.

Get input directly from a TTY -- not stdin

By default `read` will read input from stdin, but there are situations when you want to get input from the user's TTY instead of stdin. For example, say you piped output from another program into your script then it would try to read input from the user, not the pipeline (what the script now sees as stdin). Another example, you want a boot script to ask the user for input before the console TTY has been opened and attached to stdin (getty) -- this situation came up while I was building an embedded Linux system where I needed to read input from the user through the serial port (/dev/ttyS0) during boot to allow for an optional boot sequence.

In this example, technically `read` still thinks it's reading from stdin -- wWe just redirect input from a tty file.

read YN < `tty`

The `tty` command will tell you which tty you are currently logged into. The console ttys are usually on '/dev/tty[0-9]+' and the virtual ttys used for SSH logins are on '/dev/pts/[0-9]+'.

$ tty
/dev/pts/13

If you switch to a console screen (CTRL-ALT-F1 and ALT-F7 to return to X11) and then you login you will see that you now become the owner of /dev/tty1. Switch to a console and login then switch back to X11 (ALT-F7) and from a shell, you see that you now own /dev/tty1. When you logout /dev/tty1 will return to root ownership.

$ ll /dev/tty1
crw------- 1 root   root 4, 1 2009-01-04 06:02 /dev/tty1
$ ll /dev/tty1
crw------- 1 my_user tty 4, 1 2009-01-04 06:03 /dev/tty1

get yes/no input from user

YES=1
NO=0
INVALID=-1
yesno()
{
    echo -e $1
    VALID_YN=$FALSE
    YN=
    rval=
    echo -e " [y/n] \c"
    read YN
    if [ -z "$YN" ]
    then
        VALID_YN=$FALSE
        rval=$INVALID
    else
        case "$YN" in
            [Yy]*)
                VALID_YN=$TRUE
                rval=$YES
                ;;
            [Nn]*)
                VALID_YN=$TRUE
                rval=$NO
                ;;
            *)
                VALID_YN=$FALSE
                rval=$INVALID
                ;;
        esac
    fi
    if [ $rval -eq $INVALID ]
    then
        echo "Invalid Response..."
    fi
    return $rval
}

read a single character then return with no Enter required ("press any key to continue...")

The following is a discussion of `stty` command. In Bash and Korn shell you can already get a single character using `read`. The following will set the variable, CHARACTER, with a single key read from stdin: `read -r -s -n 1 CHARACTER`.

Using `stty` can get confusing because many different examples do the same things in seemingly different ways. The differences are because the `stty` command has redundant and complimentary ways of doing things. For example, `stty icanon` is the same as `stty -cbreak` and `stty raw` is the same as `stty -cooked`. Raw mode does the same thing and more as '-icanon'.

This reads a single character without echo. It works two ways. If you pass no arguments to `readc` then it will create the variable REPLY and set it to the character read from stdin. If you pass a variable name argument to `readc` then it will set the given variable name to the character read from stdin.

# This reads a single character without echo.
# "Press any key to continue..."
# If a variable name argument is given then it is set to a character read from stdin.
# else the variable REPLY is set to a character read from stdin.
# This is equivalent to `read -r -s -n 1` in Bash.
# These two examples read a single character and print it:
#     readc CHARACTER
#     echo "CHARACTER is set to ${CHARACTER}."
#     readc
#     echo "REPLY is set to ${REPLY}."
readc ()
{
    previous_stty=$(stty -g)
    stty raw -echo
    char=`dd bs=1 count=1 2>/dev/null`
    stty "${previous_stty}"

    if [ -n "$1" ] ; then
        eval $1="${char}"
    else
        REPLY="${char}"
    fi
}

The problem with read -t 0

In Bash read -t 0 is fairly useless in stdin without having previously set stty -icanon.

# Echo all data read in a tight loop for one second.
stty -echo -icanon min 0 time 0 <&0
user_data=""
timelimit=$((1 + $(date "+%s")))
while [ $(date "+%s") -lt ${timelimit} ]; do
        if read -t 0; then
                read -n 1 partial
                user_data=${user_data}${partial}
        fi
done
stty echo icanon <&0
echo ${user_data}

# Similar to the previous version, but should be more efficient.
# Echo all data input over a 1 second period of time.
stty -echo -icanon min 0 time 0 <&0
user_data=""
sleep 1
while read -t 0; do
      read -n 1 partial
      user_data=${user_data}${partial}
done
stty echo icanon <&0
echo ${user_data}

read with a timeout

I never quite got this working again. I'm pretty sure it worked, but I got interrupted when I was copying my notes. Now that I've come back to this I can't get it working again. Reading online it seems that most versions of Bash 3 and Bash 4 have a bug in the built-in read command.

The Bash built-in already has a timeout option. The following solution will work under most POSIX Bourne style shells:

read_timeout() {
        trap : USR1
        trap 'kill "${pid}" 2>/dev/null' EXIT
        (sleep "$1" && kill -USR1 "$$") &
        pid=$!
        read "$2"
        ret=$?
        kill "${pid}" 2>/dev/null
        trap - EXIT
        return "${ret}"
}

Example usage:


mktempdir -- Make Temp Directory

This is a fairly safe and fairly portable way to create a temporary directory with a unique filename. This does not clean up or delete the directory for you when done.

mktempdir () {
    CLEAN_NAME=$(echo $0 | sed -e "s/[^[:alpha:]]//g")
    NEW_TMPDIR=${TMPDIR-/tmp}/$(date "+tmp-${CLEAN_NAME}.$$.%H%M%S")
    (umask 077 && mkdir ${NEW_TMPDIR} 2>/dev/null && echo ${NEW_TMPDIR}) || return 1
    return 0
}

Use it like this:

if ! MYTEMPDIR=$(mktempdir); then
        echo "Could not create a temporary directory." >&2
        exit 1
fi

check if running as root

Check if user is root.

if [ $(id -u) -eq 0 ]; then
    echo "You are root. Good for you."
fi

Check if user is not root.

if [ $(id -u) -ne 0 ]; then 
    echo "ERROR: You must be root to run this script." >&2
    echo "Perhaps you forgot to use sudo." >&2
fi

check if process is running

Show the pids of all processes with name "openvpn":

ps -C openvpn -o pid=

Show if a process with pid=12345 is running:

kill -0 12345
echo $?

Check if a process with a given command name and pid is still running. For example, check if ssh process is running with pid 12345: "checkpid ssh 12345". Checkpid script:

#!/bin/sh
# example: checkpid ssh 12345
CMD=$1
PID=$2
for QPID in $(ps -C $CMD -o pid=); do
    if [ $QPID = $PID ]; then
        echo "running"
        exit 0
    fi
done
echo "not running"
exit 1

Bash vs. Bourne quirks

function definitions

Bash will allow function and variable definitions with dashes "-" in them. Bourne shell will not allow the following.

epoch-date ()
{
    date "+%s"
}

References

Google shell style guide