wigum::website - Workshop

Docker Workshop

2025-09-07T00:00:00+00:00

Docker and containerization have revolutionized the way applications are developed, deployed, and managed. Containers provide a lightweight and efficient way to package applications and their dependencies, ensuring consistency across different environments.

Setup

The original workshop was designed to use with a virtual machine (willGuimont/IFT2001-Docker). While you can follow the instructions below to set up the virtual machine, you can also choose to follow the workshop using Docker.

Follow these steps to prepare your environment.

Install Docker
docker run -it --rm ubuntu:latest bash
In the Docker container, run:
1. apt update && apt install -y git
2. mkdir .atelier && cd .atelier && git clone https://github.com/willGuimont/IFT2001-Docker ateliers-iftglo-2001
3. cd ateliers-iftglo-2001
4. ./helper/install.sh
5. python3 ./helper/prepare_bashrc.py && source ~/.bashrc
6. Copy the files in your home directory: cd ~/.atelier/ateliers-iftglo-2001/files && cp -r . ~/

Instructions

The instruction above sets up your environment for the workshop. The grading commands are of the form correction_nn, where nn is an integer representing the exercise number, for example correction_03 for exercise 3. Instructions for validating commands will be provided with the first question that has an expected result.

To complete this workshop, you must perform each exercise directly in the terminal. You will use the nano text editor to modify the requested scripts. To open a file with nano, run the following in the terminal.

nano test.sh

You can then edit the file; enter

echo 'Hello world'

To exit nano, press CTRL-X, then answer y to save the changes. Run the following commands to validate that everything works:

chmod +x test.sh
./test.sh

These commands should print Hello world.

Adventurers only

If you are comfortable with the command line, we encourage you to try completing this workshop using vim as your text editor. A brief introduction to vim commands is available on MIT’s The Missing Semester. vim offers many shortcuts to edit text and code very efficiently. As a programmer, you will spend a lot of time writing code, so investing in learning vim is worthwhile for the rest of your career. There are plugins emulating vim commands for most IDEs: Vim for VS Code or IdeaVim for JetBrains products.

Please note that no assistance regarding vim will be provided during this workshop.

Solutions

Proposed solutions to the exercises in these workshops are available in the following GitHub repository: ulavalIFTGLOateliers/IFT2001-Docker.

Docker

After your success with Bash, your manager has given you a new mission: find a solution to the application deployment problem. Your company frequently encounters outages due to incompatibilities between different dependency versions. Indeed, each developer can choose the operating system they wish to work on. Consequently, some developers use different Linux distributions (Ubuntu, Arch, Gentoo), while others prefer Windows.

This situation often leads to version issues, as dependencies vary from one OS to another. And worse, the versions used differ from those deployed on the production server. The recurring excuse “It worked on my machine” when a developer causes a production outage has finally exasperated your manager. He is asking you to find a solution.

That’s how you discover Docker, a platform that lets you build, deploy, and run applications in lightweight, isolated containers. With Docker, you can create a specific container for each microservice, including all required libraries and dependencies. This approach ensures that each application will run consistently, regardless of library versions used by individual developers. Moreover, Docker allows you to reproduce the development environment on the production server, thereby eliminating issues stemming from environment differences.

With Docker, you will tackle the challenge of running your company’s three main microservices. Here are the microservices you must get working:

a web server health monitoring application written in Haskell;
a document management API written in Rust, requiring a PostgreSQL database; and
a Python application for visualizing web server health.

┌────────────────────────────── MICROSERVICES (ARCH OVERVIEW) ───────────────────────────┐
│                                                                                        │
│  (3) python_app (CLI)                                                                  │
│      Image: python_app                                                                 │
│      Mode: --network=host (workshop)                                                   │
│      BACKEND_URL = http://localhost:8080                                               │
│      Uses: list/add endpoints, view uptime stats                                       │
│            │                                                                           │
│            │ HTTP (JSON)                                                               │
│            ▼                                                                           │
│  (1) status-checker (Haskell)                                                          │
│      Image: status-checker                                                             │
│      Port: 8080                                                                        │
│      Storage: monitoring.sqlite (SQLite)                                               │
│            │                                                                           │
│            │ HTTP health probe (/documents or /health)                                 │
│            ▼                                                                           │
│  (2) rust_api (Actix Web)                                                              │
│      Image: rust_api (multi-stage)                                                     │
│      Port: 8081                                                                        │
│      Env: DATABASE_URL=postgres://postgres:postgres@db                                 │
│            │                                                                           │
│            │ SQL (Diesel)                                                              │
│            ▼                                                                           │
│      PostgreSQL                                                                        │
│      Image: postgres                                                                   │
│      Port: 5432                                                                        │
│      Volume: db:/var/lib/postgresql/data                                               │
│                                                                                        │
│  Data Flows (top→down):                                                                │
│    python_app ──HTTP──▶ status-checker ──HTTP──▶ rust_api ──SQL──▶ PostgreSQL          │
│                                                                                        │
│  Summary:                                                                              │
│    - python_app only contacts status-checker.                                          │
│    - status-checker pings rust_api for availability.                                   │
│    - rust_api persists documents in Postgres.                                          │
│                                                                                        │
│  Ports: 8080 (status-checker) | 8081 (rust_api) | 5432 (Postgres)                      │
└────────────────────────────────────────────────────────────────────────────────────────┘

Why containerization?

Isolation
One of the main advantages of containerization is isolation. It often happens that different applications require specific dependency versions. With Docker, you can install different dependency versions in distinct containers, without risking conflicts or compromising other applications. Furthermore, if you need to run applications on different operating systems, such as Ubuntu and Arch Linux, you can simply wrap them in Docker containers, avoiding the need to rewrite applications for each system.

Portability
Portability is another key advantage of Docker. Once you have set up a development environment with all required dependencies, reproducing that configuration on other machines or for new team members becomes tedious. Thanks to Docker, you can create an image that contains all required dependencies and configuration, which can be easily distributed and run on different platforms, whether Linux, Windows, or others. This enables a new team member to start quickly without spending weeks configuring their environment. It’s also an advantage when deploying applications to a server.

Reproducibility
Reproducibility is also simplified with Docker. Docker images are versioned, which means it is easy to reproduce builds identically, ensuring that development, test, and production environments are consistent. You just need to specify the version of the Docker image used to guarantee consistency and avoid issues related to variations across environments.

Efficiency
In terms of efficiency, Docker offers a significant advantage over virtual machines (VMs). Unlike VMs, Docker does not need to run a full operating system for each container, which greatly reduces resource consumption. Docker containers share the host kernel, making them lightweight and quick to start.

Community
Docker’s surrounding community is very active and offers a multitude of preconfigured images for various popular tools and technologies such as Node.js, Python, and many others. These preconfigured images facilitate deployment and usage of these technologies, saving developers time by avoiding manual environment setup.

Industry
Finally, it’s important to emphasize that Docker has become a de facto standard in the software development industry. Many companies use Docker for development, testing, and deployment of their applications. Docker is now an essential tool for software developers.

What is containerization?

Docker is an open-source software platform that lets you create, deploy, and run applications in lightweight, isolated containers. A Docker container is a runtime unit that encapsulates an application together with all of its necessary elements, such as libraries, dependencies, and configuration files. These containers are self-contained and portable, meaning they can run consistently across different systems, whether a development, test, or production environment.

Docker is similar to a virtual machine that isolates an application from the host system. However, Docker is much more efficient than a VM, which requires a full operating system for each instance. Docker instead shares the host operating system’s kernel, saving resources and making containers faster to start and run.

Here is some Docker terminology. You do not have to read the advanced sections to complete this workshop. As a supplement, you can watch the video “Never install locally”.

Docker Image
An image contains the filesystem and user-space components required by the application: binaries, libraries, dependencies, configuration files, and often a Linux distribution’s userland. It does not contain its own Linux kernel. Docker images are typically created from files called Dockerfile that specify the steps to build the image.

Dockerfile
A Dockerfile is a text file that contains the instructions to build a Docker image. It specifies the image layers, dependencies to install, files to include, and commands to run during image construction.

Docker Container
A Docker container is a running instance of a Docker image. It is an isolated environment that runs the application with its dependencies. Containers are lightweight, portable, and self-contained, which makes them easy to deploy across different machines.

Docker Registry
A Docker registry is a centralized repository that stores and manages Docker images. The default public registry is Docker Hub, where you can find many ready-to-use images. You can also create and use your own private registry to store your own images.

Union filesystem (advanced concepts)
The Union Filesystem, also known as UnionFS or OverlayFS, is a technology used by Docker to manage images and container layers efficiently. The Union Filesystem allows you to overlay multiple file systems into a single logical view without physically merging them. This means Docker images and containers can share and reuse common file layers, saving storage space. It also speeds up image builds by caching layers already built. When a container starts, a new read-write layer is added on top of the image layers, enabling container-specific changes without affecting others.

cgroups (advanced concepts)
cgroups, or control groups, are a Linux kernel feature used by Docker to limit and manage system resources used by containers. cgroups allow control of resources such as CPU, memory, disk bandwidth, and network, ensuring balanced and fair resource usage among containers. Docker uses cgroups to set limits and quotas on resources allocated to each container, ensuring isolation and predictable performance.

Namespaces (advanced concepts)
Namespaces are a Linux kernel feature that isolates system resources among processes. Docker uses several types of namespaces to provide isolation between containers, including the PID namespace (process isolation), the network namespace (network isolation), the user namespace (user isolation), and the mount namespace (mount point isolation). These namespaces ensure that each container has its own isolated view of the system, preventing processes in one container from interfering with others or with the host system.

Chroot jail (advanced concepts)
Chroot, or change root, is a Unix/Linux feature that changes a process’s root directory and limits its access to the file system. Docker uses chroot to create an isolated environment inside the container, where the container’s root directory becomes the starting point for all file paths. This limits the container’s access to files and directories outside its isolated environment, thereby strengthening security and isolation.

Containers manually

To clearly illustrate what Docker does, we will manually build a simplified container without using Docker. This section is inspired by p8952/bocker: Docker implemented in around 100 lines of bash. In a terminal, enter the following commands:

# Verify that neofetch is not installed
# The following command should raise an error; do not install the package
neofetch

# Folder for containers
mkdir ~/containers; cd ~/containers

# Build the container's filesystem
sudo apt update; sudo apt install -y debootstrap
sudo debootstrap jammy ./ubuntu-container http://archive.ubuntu.com/ubuntu/

# Create an isolated environment (namespace)
sudo unshare --uts --pid --mount --ipc --fork

# Mount process, system, device, and temp folders
mount -t proc none ./ubuntu-container/proc/
mount -t sysfs none ./ubuntu-container/sys
mount -o bind /dev ./ubuntu-container/dev
mount -o bind /tmp ./ubuntu-container/tmp/
cp /etc/apt/sources.list ./ubuntu-container/etc/apt/sources.list

# Use chroot to launch a shell in the container
chroot ./ubuntu-container/ /bin/bash
# and there you go, you're in an isolated container

# Install neofetch
apt update
apt install -y neofetch

# Test neofetch; the command works!
neofetch

# You can run commands in the container here.

# Exit the container
exit

# Exit unshare
exit

# If we try neofetch again, the command is not found.
# We have indeed isolated the container!
neofetch

As you can see, there is no magic in containers. There is no magic in a container: at its core, Linux provides mechanisms such as namespaces, filesystem isolation, and cgroups. Docker packages these mechanisms behind a convenient image, networking, storage, and lifecycle-management interface.

Docker basics

Exercise 01 – Running a prebuilt image

Run the Docker image hello-world.

To run a Docker image, use the run command.
docker run image-name

If the command works, you will see a message displayed.

Solution

docker run hello-world

To execute a command in the container, specify the command at the end of run. In the following examples, we pass bash -c 'echo "Hello world"' which interprets the string in the Bash interpreter.
# Prints 'Hello world' in an ubuntu container
docker run ubuntu bash -c 'echo "Hello world"'
# Prints information about the host system
cat /etc/os-release
# Prints information from a container running Arch Linux
docker run archlinux bash -c 'cat /etc/os-release'

To launch an interactive command (such as a shell), use the -it flag. The --rm flag removes the container once it exits.
docker run -it --rm archlinux bash

Exercise 02 – Container management

To see running containers: docker ps.

To stop a container: docker stop container_id.

Note that the command may take some time to run.

Tasks:

Launch an interactive terminal with Docker;
Open another terminal and inspect running containers;
Stop the Docker container from the second terminal.

Solution

# In the first terminal
docker run -it archlinux bash

# In the second terminal
docker ps
docker stop id
# or
docker rm -f id

Exercise 03 – Inspecting a running container

Containers often run in the background. Docker provides several commands to inspect and interact with running containers.

To display the output produced by a container:
docker logs container_name
To execute a command inside a running container:
docker exec container_name command
For example, to open an interactive shell:
docker exec -it container_name bash
Finally, docker inspect displays detailed information about a Docker object, including its network configuration, mounted filesystems, environment variables, and process configuration:
docker inspect container_name

Launch an nginx container in the background:

docker run --name web-server -d -p 8080:80 nginx

Tasks:

Verify that the container is running.
Display the container’s logs.
Open an interactive shell inside the running container.
From inside the container, inspect the files in /usr/share/nginx/html.
Use docker inspect to find the container’s IP address.
Stop and remove the container.

Solution


# Verify that the container is running
docker ps

# Display its logs
docker logs web-server

# Execute Bash inside the existing container
docker exec -it web-server bash

# Inside the container
ls /usr/share/nginx/html
exit

# Inspect the container
docker inspect web-server
# Find the IP address
docker inspect \
    --format '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' \
    web-server

# Stop and remove it
docker stop web-server
docker rm web-server

docker run creates a new container, whereas docker exec executes another process inside an already running container. This distinction is useful when debugging services running in the background.

Exercise 04 – PostgreSQL database

To set environment variables in a container, use the -e flag.
docker run -it --rm -e HELLO=hello archlinux sh -c 'echo $HELLO'

To open a network port, use the -p host:docker flag, where host is the port number on the host and docker is the port number inside the container.
docker run -p 127.0.0.1:8080:80 nginx
# You can access the port via the URL http://localhost:8080/
curl http://localhost:8080/

Launch a PostgreSQL database with Docker.

Requirements:

Image name: postgres
Set the environment variable POSTGRES_PASSWORD=postgres
Map Docker port 5432 to host port 5432
Leave this container running.

Solution

docker run -e POSTGRES_PASSWORD=postgres -p 5432:5432 --rm postgres
# or
docker run --name postgres -e POSTGRES_PASSWORD=postgres -p 5432:5432 --rm -d postgres

Dockerfile

Dockerfiles are configuration files used to create custom Docker images. They allow you to define, reproducibly and automatically, an application’s runtime environment inside a Docker container.

A Dockerfile contains a series of instructions that specify the steps needed to build a Docker image. These instructions include actions such as selecting the base image, installing dependencies, configuring environment variables, copying files, running commands, and more.

Here is an annotated example Dockerfile for a Python application.

# Use the Python 3.9 slim base image
FROM python:3.9-slim

# Set the working directory inside the container
# Specifies the directory for RUN, CMD, ENTRYPOINT, COPY, and ADD
WORKDIR /app

# Copy requirements.txt from the host into the Docker container
COPY requirements.txt .

# Run a Bash command to install Python dependencies
RUN pip install -r requirements.txt

# Copy the source code into the Docker container
COPY . .

# Set the default command when the container starts
CMD ["python", "app.py"]

To build the image, use docker build -t <tag> .. The -t <tag> argument gives the image the name <tag>. For the previous application, you could use docker build -t python-app .. Note that the order of instructions matters.

It is crucial to define the order of operations in a Dockerfile due to how Docker builds images. Each instruction in the Dockerfile creates a new layer in the Docker image, and the order can significantly impact build efficiency and performance.

Docker uses a cache to speed up image builds. When you run an instruction, Docker checks whether that instruction has already been executed in a previous layer. If so and the parameters are identical, Docker reuses the cached layer instead of rebuilding it. This saves build time. However, if you modify an instruction higher up in the Dockerfile, all subsequent instructions will have their cache invalidated and must be rebuilt. For example, if you perform resource-intensive operations such as compiling code that changes whenever you modify the source, it can be preferable to place those steps towards the end of the Dockerfile to benefit from caching as much as possible.

Exercise 05 – Haskell service Dockerfile

Use the README.md in the folder ~/Applications/status-checker to write a Dockerfile that runs the Haskell application.

Requirements:

Base image haskell:9.0-buster
Run stack setup --install-ghc
Workdir /app
Copy the source code into /app
Run stack build
Launch the server with stack run

Then, start the container exposing port 8080. If everything works, you should be able to run:

curl http://localhost:8080/endpoints
# Expected response
{}%
# Or
{}

Leave this container running.

Solution

Dockerfile:

FROM haskell:9.0-buster
RUN stack setup --install-ghc
WORKDIR /app
COPY . .
RUN stack build
CMD ["stack", "run"]

Execution:

docker build -t status-checker .
docker run --rm -p 8080:8080 status-checker

Exercise 06 – Python application Dockerfile

Use the README.md in the folder ~/Applications/python_app to write a Dockerfile that runs the Python application.

Then, launch the application interactively and with the correct network mode.

This application will need to access your computer’s local network to make requests to the container from the previous exercise; to keep this exercise simple, use --network="host" when running the container.

Host networking makes the container share the host’s network stack. This is convenient here, but it reduces network isolation and is not the usual way for containers to communicate with each other. Later, Docker Compose will place the containers on a shared Docker network, where services can communicate directly using their service names.

If everything works, you should be able to use the application to add URLs to monitor and see the results.

Solution

Dockerfile:

FROM python:3.9-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install -r requirements.txt
COPY . .
CMD ["python", "main.py"]

Execution:

docker build -t python_app .
docker run --rm -it --network=host python_app

Advanced Docker

Sometimes, you may need to create more complex Docker images. Here, we will explore two advanced Docker features: multi-stage builds and volumes.

Multi-stage image builds

Multi-stage builds are an advanced Docker feature that enables optimized image creation using several distinct stages in the build process. This separates build and production stages, often resulting in final images that are lighter and more secure by minimizing what remains in the final image.

For example, for a JavaScript React application, we can split the build into two stages.

Create the application:

npx create-react-app my-app

Dockerfile:

# Stage 1: Build the application

# Use `as build` to name this build stage
FROM node:14-alpine as build
WORKDIR /app
COPY package*.json ./
RUN npm install
COPY . .
RUN npm run build

# Stage 2: Prepare the server
# Here, we start a new image from scratch
FROM nginx:alpine
# Copy the compiled app from the previous stage
COPY --from=build /app/build /usr/share/nginx/html
EXPOSE 80
CMD ["nginx", "-g", "daemon off;"]

To run:

docker build -t web-app .
docker run --rm -p 80:80 web-app

Exercise 07 – Rust multi-stage build

Use the README.md in the folder ~/Applications/rust_api to write a Dockerfile that runs the application.

Requirements:

Launch the PostgreSQL database from a previous exercise;
Create a first stage to compile the Rust code;
Create a second stage to run the code;
- Start this image from debian:bullseye-slim;
- Install the package libpq-dev with apt update; apt install -y libpq-dev;
- Set the environment variable DATABASE_URL=postgres://postgres:postgres@localhost
- Copy /app/target/release/rust_api from the previous stage;
Run the container with --network=host and expose the correct port;
Test that everything works with curl http://localhost:8081/documents.

Solution

FROM rust:latest AS build
WORKDIR /app
COPY . .
RUN cargo build --release

FROM debian:bullseye-slim
RUN apt update; apt install -y libpq-dev
WORKDIR /app
COPY --from=build /app/target/release/rust_api ./rust_api
ENV DATABASE_URL=postgres://postgres:postgres@db
CMD ./rust_api

Execution

docker run -e POSTGRES_PASSWORD=postgres -p 5432:5432 postgres
docker build -t rust_api .
docker run --rm --network=host rust_api

# Validate that the server works
curl http://localhost:8081/documents

Volumes and bind mounts

Containers have a writable filesystem, but this filesystem belongs to the container. If the container is removed, the data written inside it is also removed. Docker provides mounts to store data outside the container’s filesystem.

There are two common types of mounts: volumes and bind mounts.

A volume is storage managed by Docker. You give the volume a name, and Docker decides where the data is stored on the host:

docker run -v database-data:/var/lib/postgresql/data postgres

Here, database-data is a Docker volume and /var/lib/postgresql/data is the directory where PostgreSQL sees it inside the container. The data remains available even if the container is removed. Volumes are therefore particularly useful for persistent application data such as databases.

A bind mount directly maps an existing file or directory from the host filesystem into the container:

docker run -v ./src:/app/src my-app

Here, ./src is a directory on the host and /app/src is where it appears inside the container. Changes made on either side are immediately visible on the other side.

Volumes are generally appropriate for persistent application data. Bind mounts are useful when the container must directly access files from the host, for example source code, configuration files, or datasets.

For example, we can persist PostgreSQL data using a named volume:

docker run \
    -e POSTGRES_PASSWORD=postgres \
    -p 5432:5432 \
    --rm \
    -v postgres-data:/var/lib/postgresql/data \
    postgres

Docker creates and manages postgres-data. Removing the container does not remove the volume.

During development, we can instead use a bind mount to expose the Python application’s source code directly inside the container. This avoids rebuilding the image whenever the source code changes.

A DockerfileDev can therefore omit the source code:

FROM python:3.9-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install -r requirements.txt
# We removed the line COPY . .
CMD ["bash"]

Build and run it with:

docker build -t python_app -f DockerfileDev . 
docker run --rm -it -v .:/app python_app
# In the container
python main.py
# Modify the source code
python main.py
# Changes are reflected in the container

Thus, you can reuse the same image even if the source code has changed.

Extras

Some additional Docker features and tools that may be useful in your projects.

Cleanup

When using Docker, it is important to consider the storage used by images, containers, volumes, and other Docker artifacts. Poor storage management can lead to excessive disk usage and make Docker maintenance and resource management difficult. Docker provides several prune commands to remove unused containers and images. To clean up various Docker artifacts, the command docker system prune -a -f removes all unused resources.

Docker Compose

Docker Compose is a tool that allows you to easily define and manage multi-container applications. It simplifies deployment and orchestration of Docker containers using a simple, readable configuration file.

With Docker Compose, you can specify services, networks, volumes, and other configurations needed to run an application composed of multiple containers. You can also define dependencies among containers, environment variables, exposed ports, etc.

Docker Compose uses a YAML configuration file to describe the application’s infrastructure. This file contains sections such as services, networks, volumes, etc., where you can define the different parts of your application and their configurations.

Here is an example docker-compose.yml for the Rust application that launches both the server and the PostgreSQL database:

# List of containers to launch
services:
  # Database container
  # We find many of the same parameters as with `docker run`
  db:
    # Image to launch
    image: postgres:18
    # Restart the database if the container stops
    restart: always
    # Environment variables passed to PostgreSQL
    environment:
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: postgres
    # Store the database outside the container using a named volume
    volumes:
      - db:/var/lib/postgresql/data
    # Test whether PostgreSQL is ready to accept connections
    # Starting a container does not necessarily mean that the
    # application inside it is ready yet.
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U postgres"]
      interval: 5s
      timeout: 5s
      retries: 5
      start_period: 5s
  
  # Rust server
  api:
    # Build the Dockerfile in the directory containing compose.yaml
    build: .
    # Start the API only once the database healthcheck succeeds
    depends_on:
      db:
        condition: service_healthy
    # Map host port 8081 to container port 8081
    ports:
      - "8081:8081"
    # Containers in the same Compose application share a network.
    # Services can reach each other using their service name.
    # Therefore, the database is available at hostname `db`
    # rather than `localhost`.
    environment:
      DATABASE_URL: postgres://postgres:postgres@db

# Named volumes used by the application
volumes:
  db:
    driver: local

Run with:

docker compose up --build

Compose creates a default network for the application. The api container can therefore resolve db as a hostname and connect directly to the PostgreSQL container.

depends_on normally controls startup order only. By combining it with the database healthcheck and condition: service_healthy, Compose waits until PostgreSQL is actually ready to accept connections before starting the API.

Podman

Podman is an open-source alternative to Docker. Docker and Podman are two popular containerization tools that share similar features but differ in architecture and security approach. Docker uses a client-server architecture, where the Docker daemon runs as a separate process and Docker commands are executed via the CLI. In contrast, Podman uses a daemonless architecture, meaning it runs directly as a regular user and does not require a separate daemon process. Thus, Podman can run without special privileges, limiting potential risks associated with running as superuser. See What is podman? for more information.

Podman also offers tools to manage pods, a topic outside the scope of this workshop.

Kubernetes

Kubernetes is an open-source container orchestration system that facilitates the deployment, management, and scaling of containerized applications. Using Kubernetes offers many benefits, such as fault tolerance, easier deployments, and management and scaling of containerized applications.

Bash Workshop

2025-09-06T00:00:00+00:00

Bash is a powerful command-line interpreter that allows users to interact with their operating system through text commands. As a programmer, you will often need to use the command line to perform various tasks, such as navigating the file system, managing files and directories, and executing scripts.

Setup

The original workshop was designed to use with a virtual machine (willGuimont/IFT2001-Scripting). While you can follow the instructions below to set up the virtual machine, you can also choose to follow the workshop using Docker.

Follow these steps to prepare your environment.

Install Docker
docker run -it --rm ubuntu:latest bash
In the Docker container, run:
1. apt update && apt install -y git
2. mkdir .atelier && cd .atelier && git clone https://github.com/willGuimont/IFT2001-Scripting ateliers-iftglo-2001
3. cd ateliers-iftglo-2001
4. ./helper/install.sh
5. python3 ./helper/prepare_bashrc.py && source ~/.bashrc
6. Copy the files in your home directory: cd ~/.atelier/ateliers-iftglo-2001/files && cp -r . ~/

Instructions

nano test.sh

You can then edit the file; enter

echo 'Hello world'

To exit nano, press CTRL-X, then answer y to save the changes. Run the following commands to validate that everything works:

chmod +x test.sh
./test.sh

These commands should print Hello world.

Adventurers only

If you are comfortable with the command line, we encourage you to try completing this workshop using vim as your text editor. A brief introduction to vim commands is available on MIT’s The Missing Semester. vim offers many shortcuts to edit text and code very efficiently. As a programmer, you will spend a lot of time writing code, so investing in learning vim is worthwhile for the rest of your career. There are plugins emulating vim commands for most IDEs: Vim for VS Code or IdeaVim for JetBrains products.

Please note that no assistance regarding vim will be provided during this workshop.

Solutions

Proposed solutions to the exercises in these workshops are available in the following GitHub repository: ulavalIFTGLOateliers/IFT2001-Scripting.

Introduction to Bash

It’s a day like any other at the office. You arrive early in the morning, coffee in hand and your laptop under your arm. As you enter your department, you notice that your usually calm and relaxed boss is pacing nervously down the hallway. His pale expression and trembling hands immediately catch your attention.

“Worrying — what’s going on?” you ask, concerned.

“Production has crashed,” he replies in a trembling voice. “We’re getting hundreds of customer calls, nothing works anymore…”

Your heart races as you realize the magnitude of the problem. Your company depends on a server that hosts most of its services, and it seems something is malfunctioning. Your boss looks you straight in the eye.

“Solving this problem is your priority,” he says firmly.

He hands you a sheet of paper with the login credentials to access the server. You feel both nervous and excited at the prospect of solving this critical issue. Your boss then leads you to the server room, where an old CRT screen and a keyboard await you. He explains that it’s a Linux server edition, so without a graphical interface. You will therefore have to solve the problems using only the terminal.

┌───────────────────────────── SYSTEM DIAGNOSTIC (BOOT/RESCUE) ──────────────────────────┐
│ Host: crt-prod-01                User: username               TTY: /dev/pts/0          │
│ Kernel: 5.15.x-generic           Uptime: 00:07:13             Load: 1.42 0.97 0.53     │
│ -------------------------------------------------------------------------------------- │
│ LOG SCAN: messages.txt                                                                 │
│   Errors (raw): ................. 37  (use: grep 'Error' messages.txt)                 │
│   Unique 4xx errors: ............ pending (pipe into uniq | sed ...)                   │
│                                                                                        │
│ NEXT STEPS (YOU):                                                                      │
│   [1] Locate build log directory         (Exercise 02)                                 │
│   [2] Back up critical log file          (Exercise 03)                                 │
│   [3] Fix permissions & run diagnostics  (Exercise 04)                                 │
│   [4] Corral *.out + purge temp files    (Exercises 05–07)                             │
│   [5] Triage errors with grep/uniq/sed   (Exercises 09–12)                             │
│   [6] Hunt largest files (du/find/sort)  (Exercise 13)                                 │
│   [7] Awk your way through db.tsv        (Exercise 14)                                 │
│   [8] Script + rollback automation       (Exercises 15–16)                             │
│ -------------------------------------------------------------------------------------- │
│ TIP: Build one-liners incrementally. Example pipeline scaffold:                        │
│        cat file | grep pattern | sort | head -n 5                                      │
│                                                                                        │
│ REMEMBER:                                                                              │
│   man <cmd>    --help flags    help <builtin>    set -x (debug scripts)                │
│                                                                                        │
│ STATUS LAMPS:  [CPU OK] [DISK OK] [NET OK] [SERVICE ERROR] -> Investigate app layer    │
└────────────────────────────────────────────────────────────────────────────────────────┘
(enter the command line below…)

The command line and the manual

Open a terminal with the shortcut Ctrl+Alt+T, or press the Super key (the Windows key on the keyboard) and search for the Terminal application.

As a first step, you decide to inspect the files present on the server in order to find the log files. However, you are not sure which command to use for this.

Fortunately, the command line lets you learn how each command works. The man command allows you to read the manual page corresponding to a specific command. For example, to get the documentation for the ls command, you can use the following command:

man ls

Once you have opened the manual page, you can navigate using the arrow keys to read the content. To quit the manual page, simply press the q key.

Also, some commands accept the --help argument which displays a help message describing the arguments that can be passed to the command. For example:

ls --help

By running this command, you will get a help message detailing the different options and arguments you can use with the ls command.

Do not hesitate to use the --help argument with commands you wish to explore to obtain additional information on their usage. This can help you better understand the available features and correctly use commands in your exploration of the server.

For some commands such as cd (which is a command built into the Bash shell), you should instead use

help cd

Terminal shortcuts

Auto-completion: You can use the Tab key to get auto-completion in the terminal.

Quit a command: In a terminal, the Ctrl+C shortcut, rather than copying, terminates the execution of a command. For example, the yes command indefinitely repeats the letter y in the terminal. To quit, press Ctrl+C:
yes
# Ctrl+C to quit the execution of yes
To copy and paste, the terminal uses Ctrl+Shift+C and Ctrl+Shift+V

To close a terminal, you can use Ctrl+D.

Exercise 01 – Basic commands

This exercise aims to familiarize you with a few basic commands that will be useful throughout the workshop. To do this, use the man command and the --help argument to obtain detailed information on how each command works. In the file exercice_01.txt, briefly describe the purpose of each of the following commands. This cheat sheet will be useful throughout the workshop. You can consult it with the command cat exercice_01.txt.

Open exercice_01.txt with nano with:

nano exercice_01.txt

In another terminal (Ctrl+Alt+T to open another terminal or Ctrl+Shift+T to open another terminal in a tab), use the terminal to determine the behavior of the following commands:

Commands
`nano`
`cd`
`ls`
`cat`
`mkdir`
`rm`
`rmdir`
`mv`
`pwd`
`cp`
`chmod`
`touch`

Solution

Command	Definition
`nano`	Simple text editor
`cd`	Change current directory
`ls`	List files in a directory
`cat`	Display a file’s content
`mkdir`	Create a directory
`rm`	Delete a file or directory
`rmdir`	Delete an empty directory
`mv`	Move/rename a file
`pwd`	Show the path of the current directory
`cp`	Copy a file
`chmod`	Change file permissions
`touch`	Update a file’s timestamp, or create it if it does not exist

Use the commands listed above to explore the directory tree and find the file with the .log extension.

Here are some special paths:

. represents the current directory;

.. represents the parent directory;

~ represents the user’s home directory (/home/username/);

- represents the path of the last visited directory.

You can use these special paths with several commands, notably cd.

# Navigate into the abc/def directory
cd abc/def

# Navigate to the parent directory (abc)
cd ..

# Return to the home directory
cd
# or
cd ~

# Return to the previous directory (abc)
cd -

List the files in the current directory
Move through the different directories and try to find the file with the .log extension
Copy the absolute path (from the root of the file system /) of the directory where the log files are located into the file ~/out_02.txt. Do not add a newline at the end.

Solution

Commands to run:

ls
cd ApplicationData/output/logs
ls
pwd

Expected path: /home/glo2001/ApplicationData/output/logs

Exercise 03 – `home` directory and copy

Now that you have found the log file, it’s time to make a copy of it in your home directory.

The home directory is where a user’s personal files are stored. Each user has their own directory in /home/. If your username is username, so your home directory is located at /home/username.

There is also a shortcut to refer to the home directory: ~. Thus, each of the following commands returns you to the home directory:
# Use an absolute path
cd /home/admin
# Use tilde
cd ~
# Without arguments, cd returns to the home directory
cd

Copy the .log file into the ~/log_backup directory.

Return to your home directory;
Create a new directory named log_backup in your home directory;
Copy the .log file into the backup directory, giving it the name build_backup.log.

Solution

Commands:

cd
mkdir log_backup (typo in LaTeX was mdkdir but intent is mkdir)
cp ~/ApplicationData/output/logs/build.log ~/log_backup/build_backup.log

Exercise 04 – Permissions

Now that you have backed up the log file, it’s time to run the diagnostic script provided by your boss to analyze the system. Go back to the log directory, and try to run the script diagnostic.sh using ./diagnostic.sh.

However, you encounter a permissions error. To fix this, inspect the script permissions using the ls -l command. Then, modify the permissions to make the script executable.

chmod, short for change mode, is a command used to modify access permissions for files and directories. Permissions in bash refer to the access rights granted to users and groups to read, write, and execute files and directories. These permissions control who can perform which operations on a given file or directory. You can list file and directory permissions with ls -l.
drwxr-xr-- 1 user group 4096 Sep  7 10:00 folder
-rwxr-xr-- 1 user group  123 Sep  7 10:00 file
Permissions are generally represented by one character followed by three groups of three characters each, for a total of nine characters, displayed in a specific order:

The first character indicates whether it is a file (-) or a directory (d).

The first group of three characters represents the permissions of the file’s owner.

The second group represents the permissions of the group to which the file belongs.

The third group represents the permissions for other users.

Each group of three characters consists of the following permission types:

r (read): Read the content of the file or directory.

w (write): Modify a file’s contents; for a directory, create, delete, or rename entries inside it.

x (execute): Execute a file; for a directory, traverse it and access entries within it.

To modify these permissions, you can use the chmod command. For example, to add (+) the execute permission to a file:
chmod +x script.sh
To remove the write permission from a directory:
chmod -w folder

Steps:

Inspect the file permissions;
Use chmod with the +x argument to modify permissions;
Run diagnostic.sh.

Solution

# Check permissions
ls -l
# Change permissions
chmod +x diagnostic.sh
# Run the script
./diagnostic.sh

Exercise 05 – Wildcards

The script generated about ten .out files containing the results of the system analysis. You need to move these files into a new folder named output. Instead of moving each file manually with the command mv out_01.out output/, which would be tedious, you can use the wildcard character (*), which allows you to select multiple files at once.

The * character is interpreted by the shell before the command is executed. This is called globbing.

For example:
mv *.out output/
If the current directory contains:
a.out
b.out
notes.txt
Bash expands the command to something equivalent to:
mv a.out b.out output/
The mv command itself does not interpret *.out; Bash first replaces the pattern with the matching filenames, then launches mv. By default, * does not match filenames beginning with ..

Some common glob patterns are:
*       any sequence of characters
?       exactly one character
[abc]   one character among a, b, or c
[0-9]   one digit
For example:
ls *.txt
ls file?.txt
ls image[0-9].png
Globbing is different from regular expressions. A pattern such as *.txt is a shell glob: * matches any sequence of filename characters and . is literal. In a regular expression, . matches any single character and * repeats the preceding element. grep and sed use regular expressions; find -name uses shell-style filename patterns.

Before moving the files, you can try the command cat *. This command displays the contents of matching non-hidden files in the current directory.

However, in your case, you do not want to select all files. You can specify a specific pattern by adding a prefix or suffix to the filenames. For example, to select all .sh files, you can use the command ls *.sh. This will display the list of files with the .sh extension.

Tasks:

Create a folder named output_backup in the ~/ApplicationData/output/logs directory;
Move all files ending with .out into the folder in a single command.

Solution

# Move to the correct directory
cd ~/ApplicationData/output/logs
# Create the output folder
mkdir output_backup
# Move the files into the folder
mv *.out output_backup/

Exercise 06 – Deleting files and folders

In addition to generating .out files, the script also generated temporary .tmp files and a folder named temp. These files and this folder can be deleted.

Tasks:

Delete the files with the .tmp extension;
Delete the temp folder.

Solution

# Delete .tmp files
rm *.tmp
# Delete the folder
rm -r temp

Exercise 07 – Basic scripting

To simplify the task of moving and deleting the files generated by the diagnostic.sh script, you can create a script that will automate these actions for you.

Here is an example Bash script:
#!/usr/bin/env bash
echo 'Start of script'
ls *.sh
echo 'End of script'
In this script, the line #!/usr/bin/env bash is called a shebang (she = #, bang = !). It tells the interpreter which program should be used to run the script—in this case, Bash. It would also be possible to specify another program as the interpreter. For example, to interpret the script as Python 3, we would use the following shebang: #!/usr/bin/env python3.

Using #!/usr/bin/env bash is generally preferable to #!/bin/bash, as it looks up the location of the Bash executable in the user’s environment, making it more portable across systems.

You can create a text file with the .sh extension (for example, example.sh), copy the script above into it, then make the file executable using chmod +x example.sh. Next, you can run the script using ./example.sh.

Now, create a script that performs the following operations:

Create a script named cleanup.sh in the directory ~/ApplicationData/output/logs/;
Give the script execute permissions;
Use the commands you typed in the previous exercises to create your script:
- Create a folder named output_backup;
- Move all files ending with .out into the folder in a single command;
- Delete the files with the .tmp extension;
- Delete the temp folder.
Make sure to delete the output folder you created earlier;
Run diagnostic.sh once more, and test your script.

You can use the history command to view the list of commands you have previously executed in the terminal.

Solution

#!/usr/bin/env bash

# Create the output_backup folder
mkdir output_backup
# Move the files into the folder
mv *.out output_backup/

# Delete .tmp files
rm *.tmp
# Delete the folder
rm -r temp

Advanced commands

Now that you’ve performed some cleanup operations and become familiar with the command line, it’s time to analyze the logs and outputs of the diagnostic program. To do this, you will need to combine several commands using piping and more advanced commands. Before addressing this topic, you will explore some more advanced commands that will be useful later.

Exercise 08 – Advanced commands

As in exercise 1, use the man command and the --help argument to obtain detailed information on the commands in the following table. These commands are a bit different from those seen in exercise 1; they can take a file as input, or you can pipe the output of another command to them, which will be the subject of the next section. You can create a test file (test_file.txt) to test the commands.

In the file exercice_08.txt, briefly describe the purpose of each of the following commands. This cheat sheet will be useful throughout the workshop.

Commands
`tac`
`less`
`find`
`grep`
`sort`
`uniq`
`wc`
`head`
`tail`
`du`
`curl`
`sed`
`awk`
`kill`
`sleep`

Solution

Command	Definition
`tac`	Reverse the order of lines in a file
`less`	Text pagination
`find`	File search
`grep`	Filter lines by a pattern
`sort`	Sort lines
`uniq`	Remove adjacent duplicate lines
`wc`	Count lines, words, and characters
`head`	Display the beginning of a file
`tail`	Display the end of a file
`du`	Show file sizes
`curl`	Network request (HTTP, FTP, etc.)
`sed`	Stream editor
`awk`	Interpreter for the `awk` language
`kill`	Stop a process
`sleep`	Wait for a number of seconds

Exercise 09 – Advanced commands 1

The messages.txt file in the log folder contains the error messages that cause the system outage. Unfortunately, this file also contains a lot of logs that are not useful to you. Rather than manually reading the entire file, you decide to use the Bash commands you just discovered.

Use a command to display all lines of messages.txt that contain the string Error and copy the result into the file ~/errors.txt.

Solution

grep Error messages.txt > ~/errors.txt

Exercise 10 – Advanced commands 2

After inspecting the errors that you copied into ~/errors.txt, you realize there are many duplicates. Use a Bash command to remove duplicate lines and copy the result into ~/errors_2.txt.

uniq removes repeated adjacent lines. If identical lines are separated by other lines, uniq will not detect them.

A common pattern is therefore:
sort file.txt | uniq
sort groups identical lines together, after which uniq removes the duplicates.

Solution

sort ~/errors.txt | uniq > ~/errors_2.txt
# or
sort -u ~/errors.txt > ~/errors_2.txt

Exercise 11 – Advanced commands 3

The log file ~/errors_2.txt contains HTTP errors such as:

Error 404: File not found
Error 403: Access denied
Error 500: Internal server error

For this exercise, all client errors (4xx such as 400, 403 and 404) should be reclassified as warnings while preserving the error code.

For example:

Error 404: File not found

should become:

Warning 404: File not found

With sed -E, match ^Error (4[0-9]{2}): and replace it with Warning \1:. The \1 inserts the captured error code.

The substitution has the form: s/pattern/replacement/g

where:

s means substitute;

pattern is the regular expression to match;

replacement is the text that replaces the match;

g means replace every match on a line, rather than only the first. It is optional when a pattern can match only once per line.

In the regular expression ^Error (4[0-9]{2})::

^ matches the beginning of the line, and : after the code prevents a match on a longer number such as 4000.

4[0-9]{2} matches exactly three digits starting with 4, such as 400, 403, or 404.

The parentheses capture those three digits as group 1.

In the replacement Warning \1::

\1 inserts the text captured by group 1; the literal colon follows it.

Use a Bash command to modify the messages and copy the result into ~/errors_3.txt.

Solution

sed -E 's/^Error (4[0-9]{2}):/Warning \1:/' ~/errors_2.txt > ~/errors_3.txt

Program composition

Program composition is at the heart of the Unix philosophy. Here is an excerpt from The Art of Unix Programming describing the importance of program composition (full chapter available here).

It’s hard to avoid programming overcomplicated monoliths if none of your programs can talk to each other.

Unix tradition strongly encourages writing programs that read and write simple, textual, stream-oriented, device-independent formats. Under classic Unix, as many programs as possible are written as simple filters, which take a simple text stream on input and process it into another simple text stream on output.

Despite popular mythology, this practice is favored not because Unix programmers hate graphical user interfaces. It’s because if you don’t write programs that accept and emit simple text streams, it’s much more difficult to hook the programs together.

Text streams are to Unix tools as messages are to objects in an object-oriented setting. The simplicity of the text-stream interface enforces the encapsulation of the tools. More elaborate forms of inter-process communication, such as remote procedure calls, show a tendency to involve programs with each others’ internals too much.

To make programs composable, make them independent. A program on one end of a text stream should care as little as possible about the program on the other end. It should be made easy to replace one end with a completely different implementation without disturbing the other.

— Chapter 1. Philosophy, Rule of Composition

This program composition can be achieved using different operators, which will be the subject of the next sections.

Composition

The ; operator allows you to run one command after another on the same line.

(command1 ; command2)

For example, for exercise 7, we could rewrite the program as follows:

mkdir output; mv *.out output/; rm *.tmp; rm -r temp

The && operator runs the second command only if the first one succeeded (returns an exit code equal to zero).

(command1 && command2)

# Prints "Hello World" because both commands succeed
echo 'Hello' && echo 'World'

# We print Hello because the directory exists, so cd succeeds
cd existing_directory && echo 'Hello

# We do not print Hello because the directory does not exist, so cd fails
cd non_existent_directory && echo 'Hello'

The || operator runs the second command only if the first one failed (returns a non-zero exit code).

(command1 || command2)

# Prints only Hello, because echo 'Hello' succeeds
echo 'Hello' || echo 'World'

# We do not print Hello, because cd succeeds
cd existing_file || echo 'Hello'

# We print Hello, because cd fails
cd non_existing_file || echo 'Hello'

Pipes

The | operator passes the output of one command as input to another. This operator is also called a pipe.

(command1 | command2)

# List names ending in .txt
ls | grep '\.txt$'

# Sort the lines of a file, keep only the first 5 lines
cat file.txt | sort | head -n 5

Redirection

The > operator redirects a command’s output to a file, overwriting the file.

(command > file)

ls > file.txt

For example, it is possible to download a text file with curl:

curl https://www.ulaval.ca/ > ulaval.txt

To discard a command’s output:

ls > /dev/null 2>&1

The < operator redirects a command’s input from a file.

(command < file)

wc < file.txt

The >> operator, like >, redirects a command’s output to a file, but appends to the end of the file rather than overwriting it:

(command >> file)

ls >> file.txt

The << operator is like <, but allows you to pass multiple lines.

(command <<delim [multiple lines] delim)

cat <<EOF
abc
def
hij
EOF
# We can replace EOF (end of file) with any other string
cat <<ABC
abc
def
hij
ABC

Examples

# Find the first 5 names ending in .txt in alphabetical order
ls | grep '\.txt$' | sort | head -n 5
# 1. List the names
# 2. Keep only names ending in .txt; \ makes the dot literal, and $ matches the end of the line
# 3. Sort alphabetically
# 4. Keep only the first 5 results
# Using find for regular files; -name uses a filename glob, where the dot is already literal
find . -maxdepth 1 -type f -name '*.txt' | sort | head -n 5

# Keep only lines containing the text "warning",
# replace "warning" with "error", then write the result to output.txt
cat data.txt | grep "warning" | sed 's/warning/error/g' > output.txt
# 1. Display the contents of data.txt
# 2. Filter lines to keep only those containing "warning"
# 3. Apply a regex to replace "warning" with "error"
# 4. > writes the result to the file output.txt
# Can also just use grep
grep "warning" data.txt | sed 's/warning/error/g' > output.txt

# Generate a test file
echo -e "1\t2\n2\t3\n3\t4\n" > foo.txt
# 1. Display a formatted string
# 2. Write the result to foo.txt

# Compute the sum of each column
cat foo.txt | awk '{sum += $1; sum2 += $2} END {print sum; print sum2}' \
    | xargs echo "Sum of both columns"
# 1. Display the contents of foo.txt
# 2. Awk accumulates column sums and prints them
# 3. Pass the awk result to echo
# Or omit cat
awk '{sum += $1; sum2 += $2} END {print sum; print sum2}' foo.txt \
    | xargs echo "Sum of both columns"

We recommend trying these commands one at a time to fully understand each step of the pipeline, for example:

ls
ls | grep '\.txt$'
ls | grep '\.txt$' | sort |
ls | grep '\.txt$' | sort | head -n 5

Standard input, output, and errors

Unix programs conventionally use three standard streams:

0   stdin    standard input
1   stdout   standard output
2   stderr   standard error

stdin is where a program reads its input. stdout carries normal output, while stderr carries warnings and errors.

For example:

grep "error" server.log

This reads server.log and writes matching lines to stdout. The > operator redirects stdout to a file:

grep "error" server.log > errors.txt

This is equivalent to explicitly writing 1> because 1 refers to stdout:

grep "error" server.log 1> errors.txt

Errors can be redirected separately with 2>:

grep "error" missing-file.txt 2> errors.log

Normal output would still go to the terminal, while error messages go to errors.log. You can redirect both streams independently:

command > output.txt 2> errors.txt

To discard a stream, redirect it to /dev/null:

command > /dev/null
command 2> /dev/null

To discard both stdout and stderr:

command > /dev/null 2>&1

2>&1 sends file descriptor 2 to the destination currently used by file descriptor 1.

By default, a pipeline passes only stdout from command1 to stdin of command2:

command1 | command2

Errors from command1 still appear on the terminal.

Exercise 12 – Piping and redirection

Use piping and redirection to rewrite exercises 9, 10 and 11 as a single command.

Solution

grep Error messages.txt | sort | uniq | sed -E 's/^Error (4[0-9]{2}):/Warning \1:/' > ~/out_12.txt

Exercise 13 – File sizes

Use this command which returns the list of files and their size:

find ~/Documents -type f -exec du -sb {} +

Each line contains the size in bytes and the file name separated by a Tab character. Use the output of this command to find the five largest files. Your script must return five lines in the same format as du.

Solution

find ~/Documents -type f -exec du -sb {} + | sort -rh | head -n 5 > ~/out_13.txt
# or
find ~/Documents -type f -exec du -sb {} + | sort -h | tac | head -n 5 > ~/out_13.txt
# or
find ~/Documents -type f -exec du -sb {} + | sort -h | tail -n 5 | tac > ~/out_13.txt

Exercise 14 – Data analysis

The file ~/ApplicationData/db.tsv contains data in TSV format (TAB separated value). Format:

id	date	name	type	size
123	2023-12-25	foo	error_log	23

Use this file to find the 10 rows with the smallest size (size column), keeping only the name column. You must keep the header of the retained column, i.e. the first line of ~/out_14.txt should be name.

To accomplish this task, you can use awk, a programming language specialized for text manipulation. For example, to extract the 1st and 3rd columns, separated by a Tab:
awk '{print $1 "\t" $3}' file.txt

Solution

awk '{print $5 " " $3}' db.tsv | sort -h | head -n 11 | awk '{print $2}' > ~/out_14.txt
# or
awk '{print $5 " " $3}' db.tsv | sort -h | head -n 11 | sed 's/^[^ ]* //' > ~/out_14.txt

Scripting

Now that you are more comfortable with Bash, it’s time to create more advanced scripts.

The following examples will help you understand how to create scripts that can take arguments, use variables, and implement control structures such as conditionals and loops.

#!/usr/bin/env bash
# The script arguments are available with $n (where n is a natural number)
echo "The script name is: $0"
echo "The first argument is: $1"
echo "The second argument is: $2"
echo "All arguments: $@"
# {@:2:3} means take a slice starting at 2 (1-based) of length 3
echo "Arguments from 2 to 4: ${@:2:3}"

# Variables; note there are no spaces around the = sign
name="John"
age=25
# $age substitutes the variable’s value inside a string between double quotes
echo "My name is $name and I am $age years old."
# Environment variables
echo "The value of HOME is: $HOME"

# Substitution
current_directory=$(pwd)
# Substitutions work only for double quotes "
echo "The current directory is: $current_directory using double quotes"
# Not for single quotes '
echo 'The current directory is: $current_directory using single quotes'
# It is also possible to call a command directly inside a string
path="$(pwd)"/my/path
echo "Path: $path"

# Conditionals
if [ "$age" -ge 18 ]; then
  echo "You are an adult."
else
  echo "You are a minor."
fi

# Loops
for i in 1 2 3 4 5
do
    echo $i
done

fruits=("apple" "banana" "orange")
for fruit in "${fruits[@]}"
do
    echo "I like $fruit"
done

counter=1
while [ $counter -le 5 ]
do
    echo $counter
    ((counter++))
done

# Function
greet() {
  echo "Hello, $1!"
}

greet "Alice"

Quoting variables

When a variable is expanded without quotes, Bash may split its value into multiple arguments and expand wildcard characters in it.

Consider:

file="my report.txt"
rm $file

Bash expands the command approximately as:

rm my report.txt

So rm receives two filenames:

my
report.txt

This is usually not what we want. Quote the expansion when the value should remain a single argument:

rm "$file"

Now rm receives exactly one filename:

my report.txt

The same rule applies to most variable expansions:

cd "$directory"
cp "$source" "$destination"
kill "$server_pid"
echo "$message"

As a general rule, write "$variable" unless you specifically want Bash to split the value into words or expand wildcard characters in it.

Double quotes still allow variable and command substitution:

name="Alice"
echo "Hello $name"

Single quotes do not:

echo 'Hello $name'

This prints:

Hello $name

Exercise 15 – Test script

After all that cleaning and analysis, you decide it is wiser to just roll back to a previous version of the server until the problem is fixed. As you are not sure which version is functional, you decide to write a script that will test the server and roll back to the previous version if the test fails.

The first step is to write a script that tests whether the server is functional. Save this script in the file ~/test.sh, and give it execute permissions.

#!/usr/bin/env bash

cd ~/server

# Start the server in the background
python3 -m uvicorn main:app &
# Save the process ID (pid) of the server with $!
# $! returns the pid of the last background process started
server_pid=$!

# The `stop_server` function stops the server process
stop_server() {
  kill "$server_pid"
}

# Register the `stop_server` function to be called when this script exits
trap stop_server EXIT

# Wait for the server to be ready
sleep 1

# TODO make an HTTP request to localhost:8000

# TODO If the result is an error, exit with code 1 using `exit 1`
# $? contains the return code of the last command

Solution

#!/usr/bin/env bash

cd ~/server

# Start the server in the background
python3 -m uvicorn main:app &
# Save the process ID (pid) of the server with $!
server_pid=$!

# The `stop_server` function stops the server process
stop_server() {
  kill "$server_pid"
}

# Register the `stop_server` function to be called when this script exits
trap stop_server EXIT

# Wait for the server to be ready
sleep 1

# TODO make an HTTP request to localhost:8080
curl localhost:8000 > /dev/null 2> /dev/null

# TODO If the result is an error, exit with code 1 using `exit 1`
# $? contains the return code of the last command
if [ $? -eq 0 ]; then
  echo "Server is up"
else
  echo "Server is down"
  exit 1
fi
# Or cleaner
if curl localhost:8000 > /dev/null; then
    echo "Server is up"
else
    echo "Server is down"
    exit 1
fi

Exercise 16 – Version rollback script

Now that you have a test script, you can write a script that will roll back to the previous version of the server if the test fails. Use the given revert_last_commit function to roll back to the previous version of the server.

Complete the following script to roll back to a previous version of the server until the test script passes. Save this script in the file ~/revert.sh, and give it execute permissions.

The repository contains a known working version in its history, so repeatedly moving to the previous commit will eventually succeed.

git reset --hard discards uncommitted changes. It is safe here because the workshop repository is disposable. Be careful if you use it in your own projects!

#!/usr/bin/env bash

cd ~/server

revert_last_commit() {
  git reset HEAD~1 --hard
}

exit_status=1
# TODO write a while loop while $exit_status is not equal to 0
# At each iteration, run the test.sh script
# If the script exits with code 0, stop the script with exit 0
# Otherwise, call the `revert_last_commit` function

Solution

#!/usr/bin/env bash

cd ~/server

revert_last_commit() {
  git reset HEAD~1 --hard
}

exit_status=1
while [ "$exit_status" -ne 0 ]; do
  # Run the test script
  ~/test.sh > /dev/null 2>&1
  # Save the test script status
  exit_status=$?

  if [ "$exit_status" -ne 0 ]; then
    echo "The script returned a non-zero exit status ($exit_status). Reverting commit and retrying..."
    revert_last_commit
  else
    echo "The script returned 0 (success)."
    exit 0
  fi

  # Add a delay between retries (optional)
  sleep 1
done

Extras

`.bashrc` file

Below are optional enhancements you can add to your ~/.bashrc (or ~/.bash_profile depending on your shell startup rules) to make day‑to‑day terminal work more pleasant.

Prompt customization

Simple minimal prompt (user@host:cwd$):

PS1='\u@\h:\w$ '

Prompt with colors, Git branch and exit code indicator (status inlined, no helper function):

# Compute git branch (if any)
parse_git_branch() {
  git rev-parse --is-inside-work-tree > /dev/null 2>&1 || return
  local br
  br=$(git symbolic-ref --short HEAD 2>/dev/null || git rev-parse --short HEAD 2>/dev/null)
  [ -n "$br" ] && printf ' (%s)' "$br"
}

prompt_command() {
  local exit=$?  # exit status of previous command
  local status
  if [ $exit -eq 0 ]; then
    status='\[\e[32m\]✔\[\e[0m\]'
  else
    status="\[\e[31m\]✘ $exit\[\e[0m\]"
  fi
  local git_segment=""
  if git rev-parse --is-inside-work-tree > /dev/null 2>&1; then
    git_segment=$(parse_git_branch)
  fi
  PS1="\[\e[36m\]\u@\h\[\e[0m\]:\[\e[33m\]\w\[\e[0m\]${git_segment} ${status}\n$ "
}
PROMPT_COMMAND=prompt_command

Retro prompt:

workshop_prompt() {
  local code=$?
  local green="\[\e[32m\]" red="\[\e[31m\]" cyan="\[\e[36m\]" yellow="\[\e[33m\]" reset="\[\e[0m\]"
  local status
  if [ $code -eq 0 ]; then
    status="${green}OK${reset}"
  else
    status="${red}ERR:${code}${reset}"
  fi
  local branch=""
  if git rev-parse --is-inside-work-tree > /dev/null 2>&1; then
    b=$(git symbolic-ref --short HEAD 2>/dev/null || git rev-parse --short HEAD 2>/dev/null)
    branch=" ${yellow}${b}${reset}"
  fi
  # Add user@host and current directory like the earlier prompt example
  # Format: user@host:cwd [diag:STATUS]{ branch}
  PS1="${cyan}\u@\h${reset}:${yellow}\w${reset} ${cyan}[${status}]${reset}${branch}\n$ "
}
PROMPT_COMMAND=workshop_prompt

Safety & quality of life options

# Safer rm / mv / cp
alias rm='rm -i'
alias mv='mv -i'
alias cp='cp -i'

# Human readable sizes for common tools
alias l='ls -lh'
alias la='ls -lah'
alias lt='ls -lht'

# Grep with color
alias grep='grep --color=auto'

# Shortcuts to common folders
alias docs='cd ~/Documents'
alias dls='cd ~/Downloads'
alias desk='cd ~/Desktop'

# Some fun aliases
alias please='sudo'
alias emacs='vim'

Enable some helpful shell options:

# Append to history (do not overwrite) and share across sessions
shopt -s histappend
# Re-edit a failed history substitution rather than error
shopt -s histreedit
# Correct minor directory typos with cd
shopt -s cdspell 2>/dev/null || true

Functions

# Extract various archive formats with one command
extract() {
  [ -f "$1" ] || { echo "File not found: $1" >&2; return 1; }
  case "$1" in
    *.tar.bz2) tar xjf "$1" ;;
    *.tar.gz)  tar xzf "$1" ;;
    *.tar.xz)  tar xJf "$1" ;;
    *.tar)     tar xf  "$1" ;;
    *.tbz2)    tar xjf "$1" ;;
    *.tgz)     tar xzf "$1" ;;
    *.zip)     unzip   "$1" ;;
    *.rar)     unrar x "$1" ;;
    *.7z)      7z x    "$1" ;;
    *) echo "Don't know how to extract '$1'" >&2; return 2 ;;
  esac
}

# Quickly serve current directory over HTTP (Python 3)
serve() {
  local port=${1:-8000}
  echo "Serving on http://localhost:${port} (Ctrl+C to stop)"
  python3 -m http.server "$port"
}

`PATH`

The PATH environment variable contains a list of directories where the shell looks for executable files when you type a command. When you type a command, the shell searches through each directory in the PATH variable in order until it finds an executable file that matches the command name. The first matching executable is executed.

The value of PATH is a colon-separated list of directories. Adding directories to your PATH allows you to run executables located in those directories without specifying their full path. For example, if you have a script located in ~/.bin, you can add this directory to your PATH variable to run the script from anywhere.

export PATH="$PATH:~/.bin"

Alias git

As you will likely use git a lot, here are some useful aliases to add to your ~/.bashrc or ~/.gitconfig file. Aliases allow you to create shortcuts for long commands. You can add these aliases in your ~/.gitconfig file under the [alias] section. Here’s an example of useful git aliases from here:

[alias]
	lg1 = log --graph --abbrev-commit --decorate --format=format:'%C(bold blue)%h%C(reset) - %C(bold green)(%ar)%C(reset) %C(white)%s%C(reset) %C(dim white)- %an%C(reset)%C(bold yellow)%d%C(reset)' --all
	lg2 = log --graph --abbrev-commit --decorate --format=format:'%C(bold blue)%h%C(reset) - %C(bold cyan)%aD%C(reset) %C(bold green)(%ar)%C(reset)%C(bold yellow)%d%C(reset)%n''          %C(white)%s%C(reset) %C(dim white)- %an%C(reset)' --all
	lg = !"git lg1"
  adog = log --all --decorate --oneline --graph
	st = status
	co = commit
	com = commit -m
	coam = commit --amend
	a = add
	aa = add .
	p = push
	pu = push -u origin HEAD
	cb = checkout -b
	c = checkout
	m = merge
	f = fetch
	g = pull
	wip = commit -a -m "wip"
	wap = commit -a -m
	todo = !"git commit -a -m \"todo\" && git p"
	obs = !"git commit -a -m \"obsidian\" && git p"
	refs = !"git pull && git commit -a -m \"updated references\" && git p"
	figs = !"git pull && git commit -a -m \"updated figures\" && git p"
# In word diff, . matches each non-newline character
	wdiff = diff --word-diff-regex=.
[pull]
    ff = only

wigum::website - Workshop

Docker Workshop

Setup

Instructions

Solutions

Docker

Why containerization?

What is containerization?

Containers manually

Docker basics

Exercise 01 – Running a prebuilt image

Exercise 02 – Container management

Exercise 03 – Inspecting a running container

Exercise 04 – PostgreSQL database

Dockerfile

Exercise 05 – Haskell service Dockerfile

Exercise 06 – Python application Dockerfile

Advanced Docker

Multi-stage image builds

Exercise 07 – Rust multi-stage build

Volumes and bind mounts

Extras

Cleanup

Docker Compose

Podman

Kubernetes

Bash Workshop

Setup

Instructions

Solutions

Introduction to Bash

The command line and the manual

Exercise 01 – Basic commands

Exercise 02 – File system navigation

Exercise 03 – home directory and copy

Exercise 04 – Permissions

Exercise 05 – Wildcards

Exercise 06 – Deleting files and folders

Exercise 07 – Basic scripting

Advanced commands

Exercise 08 – Advanced commands

Exercise 09 – Advanced commands 1

Exercise 10 – Advanced commands 2

Exercise 11 – Advanced commands 3

Program composition

Composition

Pipes

Redirection

Standard input, output, and errors

Exercise 12 – Piping and redirection

Exercise 13 – File sizes

Exercise 14 – Data analysis

Scripting

Quoting variables

Exercise 15 – Test script

Exercise 16 – Version rollback script

Extras

.bashrc file

Prompt customization

Safety & quality of life options

Functions

PATH

Alias git

Exercise 03 – `home` directory and copy

`.bashrc` file

`PATH`