Bash Guide for Beginners Chapter 9. Repetitive tasks

Upon completion of this chapter, you will be able to

• Use for, while and until loops, and decide which loop fits which occasion.
• Use the break and continue Bash built-ins.
• Write scripts using the select statement.
• Write scripts that take a variable number of arguments.

The for loop

How does it work?

The for loop is the first of the three shell looping constructs. This loop allows for specification of a list of values. A list of commands is executed for each value in the list.

The syntax for this loop is:

for NAME [in LIST ]; do COMMANDS; done

If [in LIST] is not present, it is replaced with in $@ and for executes the COMMANDS once for each positional parameter that is set (see Chapter 3, "Special parameters" and Chapter 7, "Checking command line arguments").

The return status is the exit status of the last command that executes. If no commands are executed because LIST does not expand to any items, the return status is zero.

NAME can be any variable name, although i is used very often. LIST can be any list of words, strings or numbers, which can be literal or generated by any command. The COMMANDS to execute can also be any operating system commands, script, program or shell statement. The first time through the loop, NAME is set to the first item in LIST. The second time, its value is set to the second item in the list, and so on. The loop terminates when NAME has taken on each of the values from LIST and no items are left in LIST.

Examples

Using command substitution for specifying LIST items

The first is a command line example, demonstrating the use of a for loop that makes a backup copy of each .xml file. After issuing the command, it is safe to start working on your sources:

[carol@octarine ~/articles] ls *.xml
file1.xml file2.xml file3.xml

[carol@octarine ~/articles] ls *.xml > list

[carol@octarine ~/articles] for i in `cat list`; do cp "$i" "$i".bak ; done

[carol@octarine ~/articles] ls *.xml*
file1.xml file1.xml.bak file2.xml file2.xml.bak file3.xml file3.xml.bak

This one lists the files in /sbin that are just plain text files, and possibly scripts:

for i in `ls /sbin`; do file /sbin/$i | grep ASCII; done

Using the content of a variable to specify LIST items

The following is a specific application script for converting HTML files, compliant with a certain scheme, to PHP files. The conversion is done by taking out the first 25 and the last 21 lines, replacing these with two PHP tags that provide header and footer lines:

[carol@octarine ~/html] cat html2php.sh

#!/bin/bash
# specific conversion script for my html files to php
LIST="$(ls *.html)"
for i in "$LIST"; do
     NEWNAME=$(ls "$i" | sed -e 's/html/php/')
     cat beginfile > "$NEWNAME"
     cat "$i" | sed -e '1,25d' | tac | sed -e '1,21d'| tac >> "$NEWNAME"
     cat endfile >> "$NEWNAME"
done

Since we don't do a line count here, there is no way of knowing the line number from which to start deleting lines until reaching the end. The problem is solved using tac, which reverses the lines in a file.

TIP: The basename command Instead of using sed to replace the html suffix with php, it would be cleaner to use the basename command. Read the man page for more info.

Warning: Odd characters You will run into problems if the list expands to file names containing spaces and other irregular characters. A more ideal construct to obtain the list would be to use the shell's globbing feature, like this: for i in $PATHNAME/*; do commands done

The while loop

What is it?

The while construct allows for repetitive execution of a list of commands, as long as the command controlling the while loop executes successfully (exit status of zero). The syntax is:

while CONTROL-COMMAND; do CONSEQUENT-COMMANDS; done

CONTROL-COMMAND can be any command(s) that can exit with a success or failure status. The CONSEQUENT-COMMANDS can be any program, script or shell construct.

As soon as the CONTROL-COMMAND fails, the loop exits. In a script, the command following the done statement is executed.

The return status is the exit status of the last CONSEQUENT-COMMANDS command, or zero if none was executed.

Examples

Simple example using while

Here is an example for the impatient:

#!/bin/bash

# This script opens 4 terminal windows.

i="0"

while [ $i -lt 4 ]
do
xterm &
i=$[$i+1]
done

Nested while loops

The example below was written to copy pictures that are made with a webcam to a web directory. Every five minutes a picture is taken. Every hour, a new directory is created, holding the images for that hour. Every day, a new directory is created containing 24 subdirectories. The script runs in the background.

#!/bin/bash

# This script copies files from my homedirectory into the webserver directory.
# (use scp and SSH keys for a remote directory)
# A new directory is created every hour.

PICSDIR=/home/carol/pics
WEBDIR=/var/www/carol/webcam

while true; do 
	DATE=`date +%Y%m%d`
	HOUR=`date +%H`
	mkdir $WEBDIR/"$DATE"
	
	while [ $HOUR -ne "00" ]; do 
		DESTDIR=$WEBDIR/"$DATE"/"$HOUR"
		mkdir "$DESTDIR"
		mv $PICDIR/*.jpg "$DESTDIR"/
		sleep 3600
		HOUR=`date +%H`
	done
done

Note the use of the true statement. This means: continue execution until we are forcibly interrupted (with kill or Ctrl+C).

This small script can be used for simulation testing; it generates files:

#!/bin/bash

# This generates a file every 5 minutes

while true; do
touch pic-`date +%s`.jpg
sleep 300
done

Note the use of the date command to generate all kinds of file and directory names. See the man page for more.

Note: Use the system The previous example is for the sake of demonstration. Regular checks can easily be achieved using the system's cron facility. Do not forget to redirect output and errors when using scripts that are executed from your crontab!

Using keyboard input to control the while loop

This script can be interrupted by the user when a Ctrl+C sequence is entered:

#!/bin/bash

# This script provides wisdom

FORTUNE=/usr/games/fortune

while true; do
echo "On which topic do you want advice?"
cat << topics
politics
startrek
kernelnewbies
sports
bofh-excuses
magic
love
literature
drugs
education
topics

echo
echo -n "Make your choice: "
read topic
echo
echo "Free advice on the topic of $topic: "
echo
$FORTUNE $topic
echo

done

A here document is used to present the user with possible choices. And again, the true test repeats the commands from the CONSEQUENT-COMMANDS list over and over again.

Calculating an average

This script calculates the average of user input, which is tested before it is processed: if input is not within range, a message is printed. If q is pressed, the loop exits:

#!/bin/bash

# Calculate the average of a series of numbers.

SCORE="0"
AVERAGE="0"
SUM="0"
NUM="0"

while true; do

  echo -n "Enter your score [0-100%] ('q' for quit): "; read SCORE;

  if (("$SCORE" < "0"))  || (("$SCORE" > "100")); then
    echo "Be serious.  Common, try again: "
  elif [ "$SCORE" == "q" ]; then
    echo "Average rating: $AVERAGE%."
    break
  else
    SUM=$[$SUM + $SCORE]
    NUM=$[$NUM + 1]
    AVERAGE=$[$SUM / $NUM]
  fi

done

echo "Exiting."

Note how the variables in the last lines are left unquoted in order to do arithmetic.

The until loop

What is it?

The until loop is very similar to the while loop, except that the loop executes until the TEST-COMMAND executes successfully. As long as this command fails, the loop continues. The syntax is the same as for the while loop:

until TEST-COMMAND; do CONSEQUENT-COMMANDS; done

The return status is the exit status of the last command executed in the CONSEQUENT-COMMANDS list, or zero if none was executed. TEST-COMMAND can, again, be any command that can exit with a success or failure status, and CONSEQUENT-COMMANDS can be any UNIX command, script or shell construct.

As we already explained previously, the ";" may be replaced with one or more newlines wherever it appears.

Example

An improved picturesort.sh script (see Chapter 9, "Nested while loops"), which tests for available disk space. If not enough disk space is available, remove pictures from the previous months:

#!/bin/bash

# This script copies files from my homedirectory into the webserver directory.
# A new directory is created every hour.
# If the pics are taking up too much space, the oldest are removed.

while true; do 
	DISKFUL=$(df -h $WEBDIR | grep -v File | awk '{print $5 }' | cut -d "%" -f1 -)

	until [ $DISKFUL -ge "90" ]; do 

        	DATE=`date +%Y%m%d`
        	HOUR=`date +%H`
        	mkdir $WEBDIR/"$DATE"
                                                                                
        	while [ $HOUR -ne "00" ]; do
                	DESTDIR=$WEBDIR/"$DATE"/"$HOUR"
                	mkdir "$DESTDIR"
                	mv $PICDIR/*.jpg "$DESTDIR"/
                	sleep 3600
                	HOUR=`date +%H`
        	done

	DISKFULL=$(df -h $WEBDIR | grep -v File | awk '{ print $5 }' | cut -d "%" -f1 -)
	done

	TOREMOVE=$(find $WEBDIR -type d -a -mtime +30)
	for i in $TOREMOVE; do
		rm -rf "$i";
	done

done

Note the initialization of the HOUR and DISKFULL variables and the use of options with ls and date in order to obtain a correct listing for TOREMOVE.

I/O redirection and loops

Input redirection

Instead of controlling a loop by testing the result of a command or by user input, you can specify a file from which to read input that controls the loop. In such cases, read is often the controlling command. As long as input lines are fed into the loop, execution of the loop commands continues. As soon as all the input lines are read the loop exits.

Since the loop construct is considered to be one command structure (such as while TEST-COMMAND; do CONSEQUENT-COMMANDS; done), the redirection should occur after the done statement, so that it complies with the form

command < file

This kind of redirection also works with other kinds of loops.

Output redirection

In the example below, output of the find command is used as input for the read command controlling a while loop:

[carol@octarine ~/testdir] cat archiveoldstuff.sh

#!/bin/bash

# This script creates a subdirectory in the current directory, to which old
# files are moved.
# Might be something for cron (if slightly adapted) to execute weekly or 
# monthly.

ARCHIVENR=`date +%Y%m%d`
DESTDIR="$PWD/archive-$ARCHIVENR"

mkdir "$DESTDIR"

# using quotes to catch file names containing spaces, using read -d for more 
# fool-proof usage:
find "$PWD" -type f -a -mtime +5 | while read -d $'\000' file

do
gzip "$file"; mv "$file".gz "$DESTDIR"
echo "$file archived"
done

Break and continue

The break built-in

The break statement is used to exit the current loop before its normal ending. This is done when you don't know in advance how many times the loop will have to execute, for instance because it is dependent on user input.

The example below demonstrates a while loop that can be interrupted. This is a slightly improved version of the wisdom.sh script from Chapter 9, "Using keyboard input to control the while loop".

#!/bin/bash

# This script provides wisdom
# You can now exit in a decent way.

FORTUNE=/usr/games/fortune

while true; do
echo "On which topic do you want advice?"
echo "1.  politics"
echo "2.  startrek"
echo "3.  kernelnewbies"
echo "4.  sports"
echo "5.  bofh-excuses"
echo "6.  magic"
echo "7.  love"
echo "8.  literature"
echo "9.  drugs"
echo "10. education"
echo

echo -n "Enter your choice, or 0 for exit: "
read choice
echo

case $choice in
     1)
     $FORTUNE politics
     ;;
     2)
     $FORTUNE startrek
     ;;
     3)
     $FORTUNE kernelnewbies
     ;;
     4)
     echo "Sports are a waste of time, energy and money."
     echo "Go back to your keyboard."
     echo -e "\t\t\t\t -- \"Unhealthy is my middle name\" Soggie."
     ;;
     5)
     $FORTUNE bofh-excuses
     ;;
     6)
     $FORTUNE magic
     ;;
     7)
     $FORTUNE love
     ;;
     8)
     $FORTUNE literature
     ;;
     9)
     $FORTUNE drugs
     ;;
     10)
     $FORTUNE education
     ;;
     0)
     echo "OK, see you!"
     break
     ;;
     *)
     echo "That is not a valid choice, try a number from 0 to 10."
     ;;
esac  
done

Mind that break exits the loop, not the script. This can be demonstrated by adding an echo command at the end of the script. This echo will also be executed upon input that causes break to be executed (when the user types "0").

In nested loops, break allows for specification of which loop to exit. See the Bash info pages for more.

The continue built-in

The continue statement resumes iteration of an enclosing for, while, until or select loop.

When used in a for loop, the controlling variable takes on the value of the next element in the list. When used in a while or until construct, on the other hand, execution resumes with TEST-COMMAND at the top of the loop.

Continue Examples

In the following example, file names are converted to lower case. If no conversion needs to be done, a continue statement restarts execution of the loop. These commands don't eat much system resources, and most likely, similar problems can be solved using sed and awk. However, it is useful to know about this kind of construction when executing heavy jobs, that might not even be necessary when tests are inserted at the correct locations in a script, sparing system resources.

[carol@octarine ~/test] cat tolower.sh

#!/bin/bash

# This script converts all file names containing upper case characters into file# names containing only lower cases.

LIST="$(ls)"

for name in "$LIST"; do

if [[ "$name" != *[[:upper:]]* ]]; then
continue
fi

ORIG="$name"
NEW=`echo $name | tr 'A-Z' 'a-z'`

mv "$ORIG" "$NEW"
echo "new name for $ORIG is $NEW"
done

This script has at least one disadvantage: it overwrites existing files. The noclobber option to Bash is only useful when redirection occurs. The -b option to the mv command provides more security, but is only safe in case of one accidental overwrite, as is demonstrated in this test:

[carol@octarine ~/test] rm *

[carol@octarine ~/test] touch test Test TEST

[carol@octarine ~/test] bash -x tolower.sh

++ ls
+ LIST=test
Test
TEST
+ [[ test != *[[:upper:]]* ]]
+ continue
+ [[ Test != *[[:upper:]]* ]]
+ ORIG=Test
++ echo Test
++ tr A-Z a-z
+ NEW=test
+ mv -b Test test
+ echo 'new name for Test is test'
new name for Test is test
+ [[ TEST != *[[:upper:]]* ]]
+ ORIG=TEST
++ echo TEST
++ tr A-Z a-z
+ NEW=test
+ mv -b TEST test
+ echo 'new name for TEST is test'
new name for TEST is test

[carol@octarine ~/test] ls -a

./  ../  test  test~

The tr is part of the textutils package; it can perform all kinds of character transformations.

Making menus with the select built-in

General

Use of select

The select construct allows easy menu generation. The syntax is quite similar to that of the for loop:

select WORD [in LIST]; do RESPECTIVE-COMMANDS; done

LIST is expanded, generating a list of items. The expansion is printed to standard error; each item is preceded by a number. If in LIST is not present, the positional parameters are printed, as if in $@ would have been specified. LIST is only printed once.

Upon printing all the items, the PS3 prompt is printed and one line from standard input is read. If this line consists of a number corresponding to one of the items, the value of WORD is set to the name of that item. If the line is empty, the items and the PS3 prompt are displayed again. If an EOF (End Of File) character is read, the loop exits. Since most users don't have a clue which key combination is used for the EOF sequence, it is more user-friendly to have a break command as one of the items. Any other value of the read line will set WORD to be a null string.

The read line is saved in the REPLY variable.

The RESPECTIVE-COMMANDS are executed after each selection until the number representing the break is read. This exits the loop.

Examples

This is a very simple example, but as you can see, it is not very user-friendly:

[carol@octarine testdir] cat private.sh

#!/bin/bash

echo "This script can make any of the files in this directory private."
echo "Enter the number of the file you want to protect:"

select FILENAME in *;
do
     echo "You picked $FILENAME ($REPLY), it is now only accessible to you."
     chmod go-rwx "$FILENAME"
done

[carol@octarine testdir] ./private.sh

This script can make any of the files in this directory private.
Enter the number of the file you want to protect:
1) archive-20030129
2) bash
3) private.sh
#? 1
You picked archive-20030129 (1)
#?

Setting the PS3 prompt and adding a possibility to quit makes it better:

#!/bin/bash

echo "This script can make any of the files in this directory private."
echo "Enter the number of the file you want to protect:"

PS3="Your choice: "
QUIT="QUIT THIS PROGRAM - I feel safe now."
touch "$QUIT"

select FILENAME in *;
do
  case $FILENAME in
        "$QUIT")
          echo "Exiting."
          break
          ;;
        *)
          echo "You picked $FILENAME ($REPLY)"
          chmod go-rwx "$FILENAME"
          ;;
  esac
done
rm "$QUIT"

Submenus

Any statement within a select construct can be another select loop, enabling (a) submenu(s) within a menu.

By default, the PS3 variable is not changed when entering a nested select loop. If you want a different prompt in the submenu, be sure to set it at the appropriate time(s).

The shift built-in

What does it do?

The shift command is one of the Bourne shell built-ins that comes with Bash. This command takes one argument, a number. The positional parameters are shifted to the left by this number, N. The positional parameters from N+1 to $# are renamed to variable names from $1 to $# - N+1.

Say you have a command that takes 10 arguments, and N is 4, then $4 becomes $1, $5 becomes $2 and so on. $10 becomes $7 and the original $1, $2 and $3 are thrown away.

If N is zero or greater than $#, the positional parameters are not changed (the total number of arguments, see Chapter 7, "Checking command line arguments") and the command has no effect. If N is not present, it is assumed to be 1. The return status is zero unless N is greater than $# or less than zero; otherwise it is non-zero.

Examples

A shift statement is typically used when the number of arguments to a command is not known in advance, for instance when users can give as many arguments as they like. In such cases, the arguments are usually processed in a while loop with a test condition of (( $# )). This condition is true as long as the number of arguments is greater than zero. The $1 variable and the shift statement process each argument. The number of arguments is reduced each time shift is executed and eventually becomes zero, upon which the while loop exits.

The example below, cleanup.sh, uses shift statements to process each file in the list generated by find:

#!/bin/bash

# This script can clean up files that were last accessed over 365 days ago.

USAGE="Usage: $0 dir1 dir2 dir3 ... dirN"

if [ "$#" == "0" ]; then
	echo "$USAGE"
	exit 1
fi

while (( "$#" )); do

if [[ $(ls "$1") == "" ]]; then 
	echo "Empty directory, nothing to be done."
  else 
	find "$1" -type f -a -atime +365 -exec rm -i {} \;
fi

shift

done

Note: -exec vs. xargs The above find command can be replaced with the following: find options | xargs [commands_to_execute_on_found_files] The xargs command builds and executes command lines from standard input. This has the advantage that the command line is filled until the system limit is reached. Only then will the command to execute be called, in the above example this would be rm. If there are more arguments, a new command line will be used, until that one is full or until there are no more arguments. The same thing using find -exec calls on the command to execute on the found files every time a file is found. Thus, using xargs greatly speeds up your scripts and the performance of your machine.

In the next example, we modified the script from Chapter 8, "Here documents" so that it accepts multiple packages to install at once:

#!/bin/bash
if [ $# -lt 1 ]; then
        echo "Usage: $0 package(s)"
        exit 1
fi
while (($#)); do
	yum install "$1" << CONFIRM
y
CONFIRM
shift
done

Summary

In this chapter, we discussed how repetitive commands can be incorporated in loop constructs. Most common loops are built using the for, while or until statements, or a combination of these commands. The for loop executes a task a defined number of times. If you don't know how many times a command should execute, use either until or while to specify when the loop should end.

Loops can be interrupted or reiterated using the break and continue statements.

A file can be used as input for a loop using the input redirection operator, loops can also read output from commands that is fed into the loop using a pipe.

The select construct is used for printing menus in interactive scripts. Looping through the command line arguments to a script can be done using the shift statement.

Exercises

Remember: when building scripts, work in steps and test each step before incorporating it in your script.

1. Create a script that will take a (recursive) copy of files in /etc so that a beginning system administrator can edit files without fear.
2. Write a script that takes exactly one argument, a directory name. If the number of arguments is more or less than one, print a usage message. If the argument is not a directory, print another message. For the given directory, print the five biggest files and the five files that were most recently modified.
3. Can you explain why it is so important to put the variables in between double quotes in the example from Chapter 9, Output redirection?
4. Write a script similar to the one in Chapter 9, The break built-in, but think of a way of quitting after the user has executed 3 loops.
5. Think of a better solution than move -b for the script from Chapter 9, Continue Examples to prevent overwriting of existing files. For instance, test whether or not a file exists. Don't do unnecessary work!
6. Rewrite the whichdaemon.sh script from Chapter 7, "Boolean operations", so that it:
   • Prints a list of servers to check, such as Apache, the SSH server, the NTP daemon, a name daemon, a power management daemon, and so on.
   • For each choice the user can make, print some sensible information, like the name of the web server, NTP trace information, and so on.
   • Optionally, build in a possibility for users to check other servers than the ones listed. For such cases, check that at least the given process is running.
   • Review the script from Chapter 9, "Calculating an average". Note how character input other than q is processed. Rebuild this script so that it prints a message if characters are given as input.