Bash Workshop
By William Guimont-Martin • 20 minutes read •
Table of Contents
Bash is a powerful command-line interpreter that allows users to interact with their operating system through text commands. As a programmer, you will often need to use the command line to perform various tasks, such as navigating the file system, managing files and directories, and executing scripts.
Setup
The original workshop was designed to use with a virtual machine (willGuimont/IFT2001-Scripting). While you can follow the instructions below to set up the virtual machine, you can also choose to follow the workshop using Docker.
Follow these steps to prepare your environment.
- Install Docker
docker run -it --rm ubuntu:latest bash
- In the Docker container, run:
sudo apt update && sudo apt install -y git
mkdir .atelier && cd .atelier && git clone https://github.com/willGuimont/IFT2001-Scripting
cd ateliers-iftglo-2001
./helper/install.sh
python3 ./helper/prepare_bashrc.py && source ~/.bashrc
- Copy the files in your home directory:
cd ~/.atelier/ateliers-iftglo-2001/files && cp -r . ~/
Instructions
The instruction above sets up your environment for the workshop. The grading commands are of the form correction_nn
, where nn
is an integer representing the exercise number, for example correction_03
for exercise 3. Instructions for validating commands will be provided with the first question that has an expected result.
To complete this workshop, you must perform each exercise directly in the terminal. You will use the nano
text editor to modify the requested scripts. To open a file with nano, run the following in the terminal.
You can then edit the file; enter
To exit nano, press CTRL-X, then answer y
to save the changes. Run the following commands to validate that everything works:
These commands should print Hello world
.
Tip
Adventurers only
If you are comfortable with the command line, we encourage you to try completing this workshop using
vim
as your text editor. A brief introduction tovim
commands is available on MIT’s The Missing Semester.vim
offers many shortcuts to edit text and code very efficiently. As a programmer, you will spend a lot of time writing code, so investing in learningvim
is worthwhile for the rest of your career. There are plugins emulatingvim
commands for most IDEs: Vim for VS Code or IdeaVim for JetBrains products.Please note that no assistance regarding
vim
will be provided during this workshop.
Solutions
Proposed solutions to the exercises in these workshops are available in the following GitHub repository: ulavalIFTGLOateliers/IFT2001-Scripting.
Introduction to Bash
It’s a day like any other at the office. You arrive early in the morning, coffee in hand and your laptop under your arm. As you enter your department, you notice that your usually calm and relaxed boss is pacing nervously down the hallway. His pale expression and trembling hands immediately catch your attention.
“Worrying — what’s going on?” you ask, concerned.
“Production has crashed,” he replies in a trembling voice. “We’re getting hundreds of customer calls, nothing works anymore…”
Your heart races as you realize the magnitude of the problem. Your company depends on a server that hosts most of its services, and it seems something is malfunctioning. Your boss looks you straight in the eye.
“Solving this problem is your priority,” he says firmly.
He hands you a sheet of paper with the login credentials to access the server. You feel both nervous and excited at the prospect of solving this critical issue. Your boss then leads you to the server room, where an old CRT screen and a keyboard await you. He explains that it’s a Linux server edition, so without a graphical interface. You will therefore have to solve the problems using only the terminal.
The command line and the manual
Open a terminal with the shortcut Ctrl+Alt+T, or press the Super key (the Windows key on the keyboard) and search for the Terminal
application.
As a first step, you decide to inspect the files present on the server in order to find the log files. However, you are not sure which command to use for this.
Fortunately, the command line lets you learn how each command works. The man
command allows you to read the manual page corresponding to a specific command. For example, to get the documentation for the ls
command, you can use the following command:
Once you have opened the manual page, you can navigate using the arrow keys to read the content. To quit the manual page, simply press the q key.
Also, some commands accept the --help
argument which displays a help message describing the arguments that can be passed to the command. For example:
By running this command, you will get a help message detailing the different options and arguments you can use with the ls
command.
Do not hesitate to use the --help
argument with commands you wish to explore to obtain additional information on their usage. This can help you better understand the available features and correctly use commands in your exploration of the server.
For some commands such as cd
(which is a command built into the Bash shell), you should instead use
Note
Terminal shortcuts
Auto-completion: You can use the Tab key to get auto-completion in the terminal.
Quit a command: In a terminal, the Ctrl+C shortcut, rather than copying, terminates the execution of a command. For example, the
yes
command indefinitely repeats the lettery
in the terminal. To quit, press Ctrl+C:# Ctrl+C to quit the execution of yes
To copy and paste, the terminal uses Ctrl+Shift+C and Ctrl+Shift+V
To close a terminal, you can use Ctrl+D.
Exercise 01 – Basic commands
This exercise aims to familiarize you with a few basic commands that will be useful throughout the workshop. To do this, use the man
command and the --help
argument to obtain detailed information on how each command works. In the file exercice_01.txt
, briefly describe the purpose of each of the following commands. This cheat sheet will be useful throughout the workshop. You can consult it with the command cat exercice_01.txt
.
Open exercice_01.txt
with nano
with:
In another terminal (Ctrl+Alt+T to open another terminal or Ctrl+Shift+T to open another terminal in a tab), use the terminal to determine the behavior of the following commands:
Commands |
---|
nano |
cd |
ls |
cat |
mkdir |
rm |
rmdir |
mv |
pwd |
cp |
chmod |
touch |
Solution
Command | Definition |
---|---|
nano | Simple text editor |
cd | Change current directory |
ls | List files in a directory |
cat | Display a file’s content |
mkdir | Create a directory |
rm | Delete a file or directory |
rmdir | Delete an empty directory |
mv | Move/rename a file |
pwd | Show the path of the current directory |
cp | Copy a file |
chmod | Change file permissions |
touch | Update a file’s timestamp, or create it if it does not exist |
Exercise 02 – File system navigation
Use the commands listed above to explore the directory tree and find the file with the .log
extension.
Note
Here are some special paths:
.
represents the current directory;..
represents the parent directory;~
represents the user’s home directory (/home/username/
);-
represents the path of the last visited directory.
You can use these special paths with several commands, notably cd
.
# Navigate into the abc/def directory
# Navigate to the parent directory (abc)
# Return to the home directory
# or
# Return to the previous directory (abc)
- List the files in the current directory
- Move through the different directories and try to find the file with the
.log
extension - Copy the absolute path (from the root of the file system
/
) of the directory where the log files are located into the file~/out_02.txt
. Do not add a newline at the end.
Solution
Commands to run:
ls
cd ApplicationData/output/logs
ls
pwd
Expected path: /home/glo2001/ApplicationData/output/logs
Exercise 03 – home
directory and copy
Now that you have found the log file, it’s time to make a copy of it in your home
directory.
Note
The home directory is where a user’s personal files are stored. Each user has their own directory in
/home/
. If your username isusername
, so your home directory is located at/home/username
.There is also a shortcut to refer to the home directory:
~
. Thus, each of the following commands returns you to the home directory:# Use an absolute path # Use tilde # Without arguments, cd returns to the home directory
Copy the .log
file into the ~/log_backup
directory.
- Return to your
home
directory; - Create a new directory named
log_backup
in yourhome
directory; - Copy the .log file into the backup directory, giving it the name
build_backup.log
.
Solution
Commands:
cd
mkdir log_backup
(typo in LaTeX wasmdkdir
but intent is mkdir)cp ~/ApplicationData/output/logs/build.log ~/log_backup/build_backup.log
Exercise 04 – Permissions
Now that you have backed up the log file, it’s time to run the diagnostic script provided by your boss to analyze the system. Go back to the log directory, and try to run the script diagnostic.sh
using ./diagnostic.sh
.
However, you encounter a permissions error. To fix this, inspect the script permissions using the ls -l
command. Then, modify the permissions to make the script executable.
Note
chmod
, short for change mode, is a command used to modify access permissions for files and directories. Permissions in bash refer to the access rights granted to users and groups to read, write, and execute files and directories. These permissions control who can perform which operations on a given file or directory. You can list file and directory permissions withls -l
.
Permissions are generally represented by one character followed by three groups of three characters each, for a total of nine characters, displayed in a specific order:
- The first character indicates whether it is a file (
-
) or a directory (d
).- The first group of three characters represents the permissions of the file’s owner.
- The second group represents the permissions of the group to which the file belongs.
- The third group represents the permissions for other users.
Each group of three characters consists of the following permission types:
r
(read): Read the content of the file or directory.w
(write): Modify or delete the file (or the contents of the directory).x
(execute): Execute a file (or traverse a directory).To modify these permissions, you can use the
chmod
command. For example, to add (+
) the execute permission to a file:
To remove the write permission from a directory:
Steps:
- Inspect the file permissions;
- Use
chmod
with the+x
argument to modify permissions; - Run
diagnostic.sh
.
Solution
# Check permissions
# Change permissions
# Run the script
Exercise 05 – Wildcards
The script generated about ten .out
files containing the results of the system analysis. You need to move these files into a new folder named output
. Instead of moving each file manually with the command mv out_01.out output/
, which would be tedious, you can use the wildcard character (*
), which allows you to select multiple files at once.
Before moving the files, you can try the command cat *
. This command displays the contents of all files in the current directory.
However, in your case, you do not want to select all files. You can specify a specific pattern by adding a prefix or suffix to the filenames. For example, to select all .sh
files, you can use the command ls *.sh
. This will display the list of files with the .sh
extension.
Tasks:
- Create a folder named
output_backup
in the~/ApplicationData/output/logs
directory; - Move all files ending with
.out
into the folder in a single command.
Solution
# Move to the correct directory
# Create the output folder
# Move the files into the folder
Exercise 06 – Deleting files and folders
In addition to generating .out
files, the script also generated temporary .tmp
files and a folder named temp
. These files and this folder can be deleted.
Tasks:
- Delete the files with the
.tmp
extension; - Delete the
temp
folder.
Solution
# Delete .tmp files
# Delete the folder
Exercise 07 – Basic scripting
To simplify the task of moving and deleting the files generated by the diagnostic.sh
script, you can create a script that will automate these actions for you.
Note
Here is an example Bash script:
#!/usr/bin/env bash
In this script, the line
#!/usr/bin/env bash
is called a shebang (she =#
, bang =!
). It tells the interpreter which program should be used to run the script—in this case, Bash. It would also be possible to specify another program as the interpreter. For example, to interpret the script as Python 3, we would use the following shebang:#!/usr/bin/env python3
.Using
#!/usr/bin/env bash
is generally preferable to#!/bin/bash
, as it looks up the location of the Bash executable in the user’s environment, making it more portable across systems.You can create a text file with the
.sh
extension (for example,example.sh
), copy the script above into it, then make the file executable usingchmod +x example.sh
. Next, you can run the script using./example.sh
.
Now, create a script that performs the following operations:
- Create a script named
cleanup.sh
in the directory~/ApplicationData/output/logs/
; - Give the script execute permissions;
- Use the commands you typed in the previous exercises to create your script:
- Create a folder named
output_backup
; - Move all files ending with
.out
into the folder in a single command; - Delete the files with the
.tmp
extension; - Delete the
temp
folder.
- Create a folder named
- Make sure to delete the
output
folder you created earlier; - Run
diagnostic.sh
once more, and test your script.
Note
You can use the
history
command to view the list of commands you have previously executed in the terminal.
Solution
#!/usr/bin/env bash
# Create the output_backup folder
# Move the files into the folder
# Delete .tmp files
# Delete the folder
Advanced commands
Now that you’ve performed some cleanup operations and become familiar with the command line, it’s time to analyze the logs and outputs of the diagnostic program. To do this, you will need to combine several commands using piping
and more advanced commands. Before addressing this topic, you will explore some more advanced commands that will be useful later.
Exercise 08 – Advanced commands
As in exercise 1, use the man
command and the --help
argument to obtain detailed information on the commands in the following table. These commands are a bit different from those seen in exercise 1; they can take a file as input, or you can pipe
the output of another command to them, which will be the subject of the next section. You can create a test file (test_file.txt
) to test the commands.
In the file exercice_08.txt
, briefly describe the purpose of each of the following commands. This cheat sheet will be useful throughout the workshop.
Commands |
---|
tac |
less |
find |
grep |
sort |
uniq |
wc |
head |
tail |
du |
curl |
sed |
awk |
kill |
sleep |
Solution
Command | Definition |
---|---|
tac | Reverse the order of lines in a file |
less | Text pagination |
find | File search |
grep | Filter lines by a pattern |
sort | Sort lines |
uniq | Remove duplicate lines |
wc | Count lines, words, and characters |
head | Display the beginning of a file |
tail | Display the end of a file |
du | Show file sizes |
curl | Network request (HTTP, FTP, etc.) |
sed | Stream editor |
awk | Interpreter for the awk language |
kill | Stop a process |
sleep | Wait for a number of seconds |
Exercise 09 – Advanced commands 1
The messages.txt
file in the log folder contains the error messages that cause the system outage. Unfortunately, this file also contains a lot of logs that are not useful to you. Rather than manually reading the entire file, you decide to use the Bash commands you just discovered.
Use a command to display all lines of messages.txt
that contain the string Error
and copy the result into the file ~/errors.txt
.Solution
Exercise 10 – Advanced commands 2
After inspecting the errors that you copied into ~/errors.txt
, you realize there are many duplicates. Use a Bash command to remove duplicate lines and copy the result into ~/errors_2.txt
.Solution
Exercise 11 – Advanced commands 3
After inspecting the filtered errors from the file ~/errors_2.txt
, replace errors with a 400
code (400
, 403
and 404
) with warnings. With sed
, replace the text using the regular expression 'Error \(4[0-9]\+\)
with the following text: Warning \1
where \1
will copy the number captured in the regular expression.
Tip
The command will have the form
sed 's/regex1/regex2/g
.
s
indicates that it is a substitution applied globally.
Use a Bash command to modify the messages and copy the result into ~/errors_3.txt
.Solution
Program composition
Program composition is at the heart of the Unix philosophy. Here is an excerpt from The Art of Unix Programming describing the importance of program composition (full chapter available here).
It’s hard to avoid programming overcomplicated monoliths if none of your programs can talk to each other.
Unix tradition strongly encourages writing programs that read and write simple, textual, stream-oriented, device-independent formats. Under classic Unix, as many programs as possible are written as simple filters, which take a simple text stream on input and process it into another simple text stream on output.
Despite popular mythology, this practice is favored not because Unix programmers hate graphical user interfaces. It’s because if you don’t write programs that accept and emit simple text streams, it’s much more difficult to hook the programs together.
Text streams are to Unix tools as messages are to objects in an object-oriented setting. The simplicity of the text-stream interface enforces the encapsulation of the tools. More elaborate forms of inter-process communication, such as remote procedure calls, show a tendency to involve programs with each others’ internals too much.
To make programs composable, make them independent. A program on one end of a text stream should care as little as possible about the program on the other end. It should be made easy to replace one end with a completely different implementation without disturbing the other.
— Chapter 1. Philosophy, Rule of Composition
This program composition can be achieved using different operators, which will be the subject of the next sections.
Composition
The ;
operator allows you to run one command after another on the same line.
(command1 ; command2
)
For example, for exercise 7, we could rewrite the program as follows:
; ; ;
The &&
operator runs the second command only if the first one succeeded (returns an exit code equal to zero).
(command1 && command2
)
# Prints "Hello World" because both commands succeed
&&
# We print Hello because the directory exists, so cd succeeds
&&
The ||
operator runs the second command only if the first one failed (returns a non-zero exit code).
(command1 || command2
)
# Prints only Hello, because echo 'Hello' succeeds
||
# We do not print Hello, because cd succeeds
||
# We print Hello, because cd fails
||
Pipes
The |
operator passes the output of one command as input to another. This operator is also called a pipe.
(command1 | command2
)
# List files, and keep only .txt
|
# Sort the lines of a file, keep only the first 5 lines
| |
Redirection
The >
operator redirects a command’s output to a file, overwriting the file.
(command > file
)
For example, it is possible to download a text file with curl
:
To discard a command’s output:
The <
operator redirects a command’s input from a file.
(command < file
)
The >>
operator, like >
, redirects a command’s output to a file, but appends to the end of the file rather than overwriting it:
(command >> file
)
The <<
operator is like <
, but allows you to pass multiple lines.
(command <<delim [multiple lines] delim
)
# We can replace EOF (end of file) with any other string
Examples
# Find the first 5 .txt in alphabetical order
| | |
# 1. List the files
# 2. Keep only files containing .txt in their name
# 3. Sort alphabetically
# 4. Keep only the first 5 results
# Keep only lines containing the text "warning",
# replace "warning" with "error", then write the result to output.txt
| |
# 1. Display the contents of data.txt
# 2. Filter lines to keep only those containing "warning"
# 3. Apply a regex to replace "warning" with "error"
# 4. > writes the result to the file output.txt
# Generate a test file
# 1. Display a formatted string
# 2. Write the result to foo.txt
# Compute the sum of each column
| |
# 1. Display the contents of foo.txt
# 2. Awk accumulates column sums and prints them
# 3. Pass the awk result to echo
We recommend trying these commands one at a time to fully understand each step of the pipeline, for example:
|
| | |
| | |
Exercise 12 – Piping and redirection
Use piping and redirection to rewrite exercises 9, 10 and 11 as a single command.Solution
| |
Exercise 13 – File sizes
Use this command which returns the list of files and their size:
|
Each line contains the size in bytes and the file name separated by a Tab character. Use the output of this command to find the five largest files. Your script must return five lines in the same format as du
.Solution
| | |
# or
| | | |
# or
| | | |
# or
| |
Exercise 14 – Data analysis
The file ~/ApplicationData/db.tsv
contains data in TSV format (TAB separated value). Format:
id | date | name | type | size |
---|---|---|---|---|
123 | 2023-12-25 | foo | error_log | 23 |
Use this file to find the 10 rows with the smallest size (size column), keeping only the name column. You must keep the header of the retained column, i.e. the first line of ~/out_14.txt
should be name
.
Note
To accomplish this task, you can use
awk
, a programming language specialized for text manipulation. For example, to extract the 1st and 3rd columns, separated by a Tab:
Solution
| | |
# or
| | |
Scripting
Now that you are more comfortable with Bash, it’s time to create more advanced scripts.
The following examples will help you understand how to create scripts that can take arguments, use variables, and implement control structures such as conditionals and loops.
#!/usr/bin/env bash
# The script arguments are available with $n (where n is a natural number)
# {@:2:3} means take a slice starting at 2 (1-based) of length 3
# Variables; note there are no spaces around the = sign
name="John"
age=25
# $age substitutes the variable’s value inside a string between double quotes
# Environment variables
# Substitution
current_directory=
# Substitutions work only for double quotes "
# Not for single quotes '
# It is also possible to call a command directly inside a string
path=""/my/path
# Conditionals
if [; then
else
fi
# Loops
for
do
done
fruits=("apple" "banana" "orange")
for
do
done
counter=1
while [
do
done
# Function
Exercise 15 – Test script
After all that cleaning and analysis, you decide it is wiser to just roll back to a previous version of the server until the problem is fixed. As you are not sure which version is functional, you decide to write a script that will test the server and roll back to the previous version if the test fails.
The first step is to write a script that tests whether the server is functional. Save this script in the file ~/test.sh
, and give it execute permissions.
#!/usr/bin/env bash
# Start the server in the background
&
# Save the process ID (pid) of the server with $!
# $! returns the pid of the last background process started
server_pid=
# The `stop_server` function stops the server process
# Register the `stop_server` function to be called when this script exits
# Wait for the server to be ready
# TODO make an HTTP request to localhost:8000
# TODO If the result is an error, exit with code 1 using `exit 1`
# $? contains the return code of the last command
Solution
#!/usr/bin/env bash
# Start the server in the background
&
# Save the process ID (pid) of the server with $!
server_pid=
# The `stop_server` function stops the server process
# Register the `stop_server` function to be called when this script exits
# Wait for the server to be ready
# TODO make an HTTP request to localhost:8080
# TODO If the result is an error, exit with code 1 using `exit 1`
# $? contains the return code of the last command
if [; then
else
fi
Exercise 16 – Version rollback script
Now that you have a test script, you can write a script that will roll back to the previous version of the server if the test fails. Use the given revert_last_commit
function to roll back to the previous version of the server.
Complete the following script to roll back to a previous version of the server until the test script passes. Save this script in the file ~/revert.sh
, and give it execute permissions.
#!/usr/bin/env bash
exit_status=1
# TODO write a while loop while $exit_status is not equal to 0
# At each iteration, run the test.sh script
# If the script exits with code 0, stop the script with exit 0
# Otherwise, call the `revert_last_commit` function
Solution
#!/usr/bin/env bash
exit_status=1
while [; do
# Run the test script
# Save the test script status
exit_status=
if [; then
else
fi
# Add a delay between retries (optional)
done
Extras
.bashrc
file
Below are optional enhancements you can add to your ~/.bashrc
(or ~/.bash_profile
depending on your shell startup rules) to make day‑to‑day terminal work more pleasant.
Prompt customization
Simple minimal prompt (user@host:cwd$):
PS1='\u@\h:\w$ '
Prompt with colors, Git branch and exit code indicator (status inlined, no helper function):
# Compute git branch (if any)
PROMPT_COMMAND=prompt_command
Retro prompt:
PROMPT_COMMAND=workshop_prompt
Safety & quality of life options
# Safer rm / mv / cp
# Human readable sizes for common tools
# Grep with color
# Shortcuts to common folders
# Some fun aliases
Enable some helpful shell options:
# Append to history (do not overwrite) and share across sessions
# Re-edit a failed history substitution rather than error
# Correct minor directory typos with cd
||
Functions
# Extract various archive formats with one command
# Quickly serve current directory over HTTP (Python 3)
PATH
The PATH
environment variable contains a list of directories where the shell looks for executable files when you type a command. When you type a command, the shell searches through each directory in the PATH
variable in order until it finds an executable file that matches the command name. The first matching executable is executed.
The value of PATH
is a colon-separated list of directories. Adding directories to your PATH
allows you to run executables located in those directories without specifying their full path. For example, if you have a script located in ~/.bin
, you can add this directory to your PATH
variable to run the script from anywhere.
Alias git
As you will likely use git
a lot, here are some useful aliases to add to your ~/.bashrc
or ~/.gitconfig
file. Aliases allow you to create shortcuts for long commands. You can add these aliases in your ~/.gitconfig
file under the [alias]
section. Here’s an example of useful git
aliases from here:
[alias]
--graph --abbrev-commit --decorate --format=format:'%C(bold blue)%h%C(reset) - %C(bold green)(%ar)%C(reset) %C(white)%s%C(reset) %C(dim white)- %an%C(reset)%C(bold yellow)%d%C(reset)' --all
log --graph --abbrev-commit --decorate --format=format:'%C(bold blue)%h%C(reset) - %C(bold cyan)%aD%C(reset) %C(bold green)(%ar)%C(reset)%C(bold yellow)%d%C(reset)%n'' %C(white)%s%C(reset) %C(dim white)- %an%C(reset)' --all
log !"git lg1"
--all --decorate --oneline --graph
log status
commit
-m
commit --amend
commit add
.
add push
-u origin HEAD
push -b
checkout checkout
merge
fetch
pull
-a -m "wip"
commit -a -m
commit !"git commit -a -m \"todo\" && git p"
!"git commit -a -m \"obsidian\" && git p"
!"git pull && git commit -a -m \"updated references\" && git p"
!"git pull && git commit -a -m \"updated figures\" && git p"
--word-diff-regex=.
diff [pull]
only