CSc 250: Lecture Notes: variables, streams, scripting

Review

Last time, we covered a bunch of new bash commands. In this lecture, we are going to switch gears a little bit. We are still going to be talking about bash, but we will be focussing on variables, standard inputs and outputs, output redirection, and scripting.

bash variables

In bash, it is possible to store values into variables.

Storing values into variables is useful for lots of things. One particular use case is to save commonly-written values so that you don’t have to repeat yourself all of the time.

Creating a bash variable is rather simple. The format is NAME_OF_VARIABLE="value of variable". A few examples:

$ NAME="Billy Joe Smith"
$ ADDRESS="1550 N Ridgetop Drive"
$ BIO="I am a person who like to write code"

In the above example, we set three variables, NAME, ADDRESS, and BIO. Variables do not need to be in upper case, but in bash making variables upper-case is a generally accepted standard, and considered good practice. Now that we have set these standards, how do we use them? First, we could try printing them with the echo command:

$ echo NAME
NAME

What, what? When we try to print out NAME all that we get is NAME in the output. Why is that? When using a variable, the $ symbol has to be pre-pended to the variable name. Also, the name of the variable should be surrounded by curly-brackets {}. The curly brackets are actually not a strict requirement, but it is good practice to use them. This indicates to bash that a variable is being used, and to treat the next character as a variable name, rather than just regular text. So let’s try that again:

$ echo ${NAME}
Billy Joe Smith

There we go! Now, try combining some variables in the same echo command:

 $ echo My name is ${NAME} and I live at ${ADDRESS}
 My name is Billy Joe Smith and I live at 1550 N Ridgetop Drive

As can be seen, we used multiple variables in a single invocation of echo.

What if we create a variable, and later want to add something to it? Let’s try:

$ GROCERY_LIST=Milk,Butter,Salt
$ echo ${GROCERY_LIST}
Milk,Butter,Salt
$ GROCERY_LIST+=,Onions,Lettuce,Bacon
$ echo ${GROCERY_LIST}
Milk,Butter,Salt,Onions,Lettuce,Bacon

In the example above, += was used to append more text to a variable that already exists. Variables can also be completely overridden:

$ FAVORITE_SPORTS_TEAM=PhoenixSuns
$ echo ${FAVORITE_SPORTS_TEAM}
PhoenixSuns
$ FAVORITE_SPORTS_TEAM=GoldenStateWarriors
$ echo ${FAVORITE_SPORTS_TEAM}
GoldenStateWarriors

standard output, error

Each running process (program) running on a UNIX system has three “standard” streams of input and output: standard output, standard error, and standard input.

As it’s name suggests, standard output (also often referred to as stdout) is the “standard” stream of output, where most of the “printing” happens. For example, when we echo Hello World, Hellow World is printed to stdout. When we cat names.txt, the content of namex.txt is printed to stdout. By default in bash, all standard output is printed to the console after running the program.

Standard error (also referred to as stderr) behaves similarly to stdout. Typically, this stream of output is where error/warning messages should be printed to. We won’t be using stderr much in this class, but it is good for you to be aware of because you will likely read about it when looking up documentation for bash and bash tools.

As mentioned, stdout and stderr are printed to the console by default. However, we can change where stdout is sent to using >. Here is an example:

$ echo "one two three"
one two three
$ echo "one two three" > stuff.txt
$ cat stuff.txt 
one two three

By typing > stuff.txt after the echo command, we are telling bash to send the stdout from the echo command to a file named stuff.txt. We can also over-write an existing file with this same method:

$ echo "one two three" > stuff.txt
$ cat stuff.txt 
one two three
$ echo "hello world" > stuff.txt
$ cat stuff.txt 
hello world

We can also append to an existing file if >> is used instead of >:

$ echo "one two three" > stuff.txt
$ cat stuff.txt 
one two three
$ echo "hello world" >> stuff.txt
$ cat stuff.txt 
one two three
hello world
$ echo "how are you?" >> stuff.txt
$ cat stuff.txt 
one two three
hello world
how are you?

standard input

Just like how stdout is a stream of text that a bash command prints out (actually, it is just a stream of bytes, but for the purposes of this class, we’ll say text), standard input (stdin) is a stream of text that bash programs recieve form the user and/or other programs.

In many of our examples so far, we have executed shell programs that operate on a file. For example: cat fine.txt, grep SearchTerm file.txt, and sort file.txt. Instead of operating on a file, many of these programs can do their “work” of sorting, searching, etc on a stream of text coming from standard input.

So how does this work? How can we tell a program like sort or grep to do it’s job with stdin instead of a text file? The easiest way to learn is by example, so let’s jump in.

The special token | (called a pipe) can be used to send the stdout form one program to the stdin of another program. It can be placed in-between commands on the command line.

Here is a command to simply print out rhe contents of a file:

$ cat file.txt 
DDD
EEE
AAA
CCC
BBB

To sort this data using pipes, we can do:

$ cat file.txt | sort
AAA
BBB
CCC
DDD
EEE

We can chain together any number of commands using pipes. We could also do a search using grep after running the sort:

$ cat file.txt | sort | grep C
CCC

Having the ability to read text form stdin has many uses, but the primary use when using bash is to be able to chain commands together like in the example above. It is completely legal to use both | and > when chaining together commands, like so:

$ cat file.txt | sort | grep C > output.txt
$ cat output.txt 
CCC

Scripting

Without going into too much detail, a bash script is a text file with a sequence of bash commands, that can be executed by simply executing the file containing the commands. Writing bash scripts us useful for many purposes, but at the very least is saves the user from having to memorize and re-type the same sequence of commands over and over again. As you might imagine, storing a sequence of commands to run is a very useful feature of bash.

How does one go about writing a bash script? First create a file with .sh as te extension. .sh is the standard extension for all bash scripts. For example, we will create a script called getDirInfo.sh, which prints information about the current working directory.

The first line of every bash script should look like this:

#!/bin/bash

The #! (called the “sha-bang”) at the beginning of a script tells your system that this file is a set of commands to be fed to the command interpreter indicated. The #! is actually a two-byte magic number, a special marker that designates a file type, or in this case an executable shell script. Typing /bin/bash after #! tells the system to use bash to interpret the commands in the file. This is why all bash commands should start with this line.

Now, let’s finish the script to print information about the current working directory:

 1  #!/bin/bash
 2   
 3  echo ""
 4  echo "The current working directory is:"
 5  pwd
 6  
 7  echo ""
 8  echo "The files in this directory are:"
 9  ls
10   
11  echo ""
12  echo "The size of the current directory is:"
13  du -sh `pwd`

Executing the script by running ./getDirInfo.sh. The output will look something like:

$ ./getDirInfo.sh 

The current working directory is:
/Users/bddicken/test

The files in this directory are:
file.txt    grades.txt  names.txt   stuff.txt
getDirInfo.sh   image.jpg   output.txt

The size of the current directory is:
 60K    /Users/bddicken/test

Let’s walk through what this script is doing, line-by-line.

This script can now be executed at any time, and saves a user from re-typing the same series of commands when this information is needed. Let’s look at another example.

We will write another script named dirs.sh:

 1 #!/bin/bash
 2  
 3 echo "----------------------------------------"
 4 echo "The files in Your current directory are:"
 5 ls
 6  
 7 echo "----------------------------------------"
 8 echo "The files one dir up are:"
 9 cd ..
10 ls
11  
12 echo "----------------------------------------"
13 echo "The files two dirs up are:"
14 cd ..
15 ls

When I run this script (./dirs.sh) from /Users/bddicken/test, It prints:

$ ./dirs.sh
----------------------------------------
The files in Your current directory are:
dirs.sh
----------------------------------------
The files one dir up are:
Applications    Documents   Google Drive    Movies      Pictures    dev     test
Desktop     Downloads   Library     Music       Public      rc
----------------------------------------
The files two dirs up are:
Shared      bddicken
$

Again, walking through what this script is doing line-by-line:

Scripting with arguments!

As we all know by now, when we use bash commands we can usually pass arguments to those commands, such as file names and string to search for. Bash scripts have a very convenient way of processing command-line arguments for us to use in our scripts.

The elements used on the command-line to execute a script are accesible via default bash variables. The bash variable $# informs us how many total options were provided on the command-line. The value of the variable ${0} is the name of the script executed. The value of the variables${1}, ${2}, ${3}, … have the values of the first, second, third, … options.

Here is a script named args.sh demonstrating how these variables can be used:

1 #!/bin/bash
2  
3 echo "The script being run is named: ${0}"
4 echo "Arguments provided:            $#"
5  
6 echo "     The first argument is:    ${1}"
7 echo "     The second argument is:   ${2}"
8 echo "     The third argument is:    ${3}"

Here are a few executions of this script:

$ ./args.sh Aye Bee See
The script being run is named: ./args.sh
Arguments provided:            3
     The first argument is:    Aye
     The second argument is:   Bee
     The third argument is:    See
$
$ ./args.sh Aye
The script being run is named: ./args.sh
Arguments provided:            1
     The first argument is:    Aye
     The second argument is:   
     The third argument is:    
$
$ ./args.sh A B C D E F G H I J K L M N O P
The script being run is named: ./args.sh
Arguments provided:            16
     The first argument is:    A
     The second argument is:   B
     The third argument is:    C

Try writing a similar script on your own, and playing around with these option variables.

As you can probably image, there are lots of useful applications of being able to access the command line options. Here’s a silly example… an MadLib script:

 1 #!/bin/bash
 2  
 3 echo "----------------------------------------------"
 4 echo "This is an MadLib script that takes five words: name1, name2, adjective, adverb, place."
 5 echo ""
 6  
 7 NAME1=${1}
 8 NAME2=${2}
 9 ADJECTIVE=${3}
10 ADVERB=${4}
11 PLACE=${5}
12  
13 echo "----------------------------------------------"
14 echo "${NAME1}: Hey ${NAME2}! How goes it?"
15 echo "${NAME2}: Good. You wanna go to ${PLACE} and play some ${ADJECTIVE} video games?"
16 echo "${NAME1}: Sure. But first I need to ${ADVERB} run some errands."
17 echo "${NAME2}: OK, see you at ${PLACE} in a bit!"

Here are a few runs of it:

$ ./madlib-story.sh Danny-T Esteban poopy speedily Spain
----------------------------------------------
This is an MadLib script that takes five words: name1, name2, adjective, adverb, place.

----------------------------------------------
Danny-T: Hey Esteban! How goes it?
Esteban: Good. You wanna go to Spain and play some poopy video games?
Danny-T: Sure. But first I need to speedily run some errands.
Esteban: OK, see you at Spain in a bit!

$ ./madlib-story.sh Bo Rex crazy slowly Madagascar
----------------------------------------------
This is an MadLib script that takes five words: name1, name2, adjective, adverb, place.

----------------------------------------------
Bo: Hey Rex! How goes it?
Rex: Good. You wanna go to Madagascar and play some crazy video games?
Bo: Sure. But first I need to slowly run some errands.
Rex: OK, see you at Madagascar in a bit!

Depending on how you create your .sh file, it might not be executable using ./my-script-name.sh by default. This is because when new files are created, they typically do not have permissions to be “executed” by a user. For example, when using touch script.sh, a file named script.sh will be created. By default, touch creates a file that the user has read and write permissions for, but not execute permissions. You can read up on file permissions on Wikipedia to get familiar with how all of this works. We won’t be covering the details of this in this class, but you should know the basics.

If bash will not execute your script when you type ./script.sh, just run the following command:

chmod +x script.sh

This will add the executable permissions to the file, and you should be able to execute it. chmod is a command that can be used for changing and managing the permissions on files and directories on a UNIX machine. the +x indicated that executable permissions should be added to the file with the provided name.