Shell Scripts

The content of this page is mainly based on MIT Missing Semester.

What we will used for this chapter is bash.

Variables

Assigning to a variable:

foo=bar
echo $foo

bar

Spaces are important. For example, this is not ok:

foo = bar

Quotes

Single quotes and double quotes.

Things in single quotes will be used "as is". However, variables and some other things in double quotes will be evaluated and replaced.

echo "Value is $foo"
echo 'Value is $foo'

Value is bar
Value is $foo

Functions

Here is a simple function definition:

# File: mcd.sh
# Here $1 means the first argument
mcd () {
    mkdir -p "$1"
    cd "$1"
}

To call it:

source mcd.sh
mcd test

Special Variables

$?: The return code of the previous command (where 0 usually means everything went fine) $_: The last argument of the previous command !!: The whole previous command (useful in sudo !!)

Examples for $?:

echo "Hello"
echo $?

Hello
0

grep foobar mcd.sh
echo $?

true
echo $?

false
echo $?

Exit Code as Condition

An or operator || can be used: if the left hand side fails (return not 0), the right hand side will be executed.

false || echo "Oops fail"

Oops fail

true || echo "Will not be printed"

Note that true always returns 0 while false always returns 1.

An and operator && can be used: if the left hand side succeeds (return 0), the right hand side will be executed.

An ; can be used: no matter whether the left hand side succeeds, the right hand side will always be executed.

Passing Command Results

Using the result of a command directly:

foo=$(pwd)
echo "We are in $(pwd)"

Passing the result of a command as a file:

cat <(ls) <(ls ..)

mkdir foo/x bar/y
diff <(ls foo) <(ls bar)

An example script:

# File: example.sh
#!/bin/bash

echo "Starting program at $(date)"
echo "Running program $0 with $# arguments with pid $$"

for file in "$@"; do
    grep foobar "$file" > /dev/null 2> /dev/null
    if [[ "$?" -ne 0 ]]; then
        echo "File $file does not have any foobar, adding one"
        echo "# foobar" >> "$file"
    fi
done

$0: the name of the script
$#: the number of arguments
$$: the process id
$@: all the arguments
writing things to /dev/null just means disgard them
> deals with the standard output
2> deals with the standard error
[ ... ] or test make judgements.
In bash, use [[ ... ]] to replace [] to prevent errors, although this keyword is not supported by sh
-ne means not equal
man test gives you the conditions you can use

The script can be run like:

./example.sh mcd.sh script.py example.sh

Globbing

* expands to any number (including 0) of characters ? expands to one character {} gives a group and will be expanded for each item in it.

ls *.sh
ls project?
convert image.{png,jpg}
touch foo{,1,2,10}
touch proj{1,2}/src/test/test{1,2,3}.py
mkdir foo bar
touch {foo,bar}/{a..j}

Shebang and Others

You can also write scripts look like shell scripts in some other programming languages. For example python.

For the scripts below, you can run them with ./script.py directly without specifing python because of the shebang:

#!/usr/local/bin/python
# Use a specific program
import sys
for arg in reversed(sys.argv[1:]):
    print(arg)

#!/usr/bin/env python
# Use the python in the environment
import sys
for arg in reversed(sys.argv[1:]):
    print(arg)

The second one is better, because env will use the python in $PATH to run the script. For the first one, the path of python cannot be guaranteed.

Using shebang (e.g. #!/bin/bash) is important and recommended, because the script may use any languages.

A function will be executed in the shell, while a script will be executed in a seperate process.

Shell Tools

To check syntax and see warnings

shellcheck mcd.sh

To check how to use a command, man and tldr are both useful. The former is usually longer, while the latter provides some examples and is more convenient to find a correct option.

To find files

find . -name src -type d
find . -path '**/text/*.py' -type f

You can also do something after the files are found:

find . -name "*.tmp" -exec rm {} \;

fd also searches for files, and it is more simple. You just use fd file_name

You can also use locate to locate things in the whole file system. However, it only supports searching with file name, and the results may not be up to date.

locate missing-semester

To search for code

Search all foobar in a directory:

grep -R foobar .

You can use -v to invert the results selected, or -C 5 to see a content of 5 lines.

rg (ripgrep) are also useful for this. They support various arguments. Example:

rg -u --files-without-match "^#\!" -t sh

The command above searches for all .sh files that do not have #! at the beggining of any line.

The idea is: You should know that using suitable tools can help you solve a problem efficiently, while which tool you use is not so important.

Finding shell commands

You can use the arrow key to navigate among the commands that you've used.

Another way to get the command history:

history
history | grep convert

A useful command fzf

history | fzf

To see a structure of directories

tree
broot
nnn

Exercise

(1)

ls -a   # list all files
ls -s -h    # with size, in human-readable format
ls -c -lt   # include last modification time, newest first

(2)

#!/bin/bash
marco() {
    export MARCO=$(pwd)
}
polo() {
    cd "$MARCO"
}