GNOME Shell Extensions & CI: Part III

In the last part of the series, I explain how I set up continuous integration tests using podman and GitHub Actions.

If you haven’t read the first parts yet, I would suggest doing this before. Here are links to the other parts:

  1. Bundling the Extension
  2. Automated Release Publishing
  3. Automated Tests with GitHub Actions (this post)

GNOME Shell in a Container

So the idea is to run GNOME Shell in a container, install the extension, and perform various tests on it. For this purpose, I created several Fedora-based containers, one for each GNOME Shell version I want to run tests on. These containers are currently available:

So here’s an example what you can do with these containers (you will need to have podman and imagemagick installed). Run the following commands one by one. The first command will download and run a container based on Fedora 33. You can then use the two other commands to run GNOME Shell, and open the gnome-control-center inside the container.

1
2
3
4
5
6
7
8
9
# Run the container in interactive mode.
podman run --rm --cap-add=SYS_NICE --cap-add=IPC_LOCK \
            -ti ghcr.io/schneegans/gnome-shell-pod-33

# Now do this inside the container to start GNOME Shell.
systemctl --user start "gnome-xsession@:99"

# For example, you can now run this command.
DISPLAY=:99 gnome-control-center

Did it work? Well, we will see! Open up another terminal on your host and execute the following commands. These will capture a screenshot of GNOME Shell inside the container!

1
2
3
4
5
6
7
8
9
# Copy the framebuffer of xvfb.
podman cp $(podman ps -q -n 1):/opt/Xvfb_screen0 .

# Convert it to jpeg (this step requires imagemagick).
convert xwd:Xvfb_screen0 capture.jpg

# And finally display the image.
# This way we can see that GNOME Shell is actually up and running!
eog capture.jpg

GNOME Shell running in a container.

I think you see where this is going. In the next example, we will perform the same steps, but with a non-interactive container. You can copy-paste the entire block below to your terminal and execute it all together.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
# Run the container in detached mode.
POD=$(podman run --rm --cap-add=SYS_NICE --cap-add=IPC_LOCK \
                  -td ghcr.io/schneegans/gnome-shell-pod-33)

# This method is used to run arbitrary commands inside the running container.
# The set-env.sh script is contained in the container image and sets all
# environment variables required to interact with the D-Bus.
# You can look at the script here:
# https://github.com/Schneegans/gnome-shell-pod/tree/master/bin
do_in_pod() {
  podman exec --user gnomeshell --workdir /home/gnomeshell \
              "${POD}" set-env.sh "$@"
}

# Wait until the user bus is available. This is also a custom script
# contained in the container.
do_in_pod wait-user-bus.sh 

# Start GNOME Shell.
do_in_pod systemctl --user start "gnome-xsession@:99"

# Wait some time until GNOME Shell has been started.
sleep 3

# Run the application.
do_in_pod gnome-control-center &

# Wait another few seconds.
sleep 3

# Now make a screenshot and show it!
podman cp ${POD}:/opt/Xvfb_screen0 . && \
       convert xwd:Xvfb_screen0 capture.jpg && \
       eog capture.jpg

# Now we can stop the container again.
podman stop ${POD}

Feel free to replace the gnome-shell-pod-33 with any other container image name. For example, gnome-shell-pod-36 will give you GNOME Shell 42 (we will have to disable this welcome tour later…):

GNOME Shell 42 running in a container.

Executing Tests in the Container

We can now use this setup to run automated tests inside the containers. The following example script uses podman cp to copy the extension zip into the running container. It then installs and enables the extension with gnome-extensions install and gnome-extensions enable respectively. Thereafter, it launches GNOME Shell and closes the initial overview & welcome tour of GNOME 40+. Finally, it opens the preferences window of the extension. To test whether this worked, the virtual screen is searched for a sub-image of the preferences dialog. For this, you can make a screen shot of a small portion of your preferences dialog, and save it as references/preferences.png to the extension’s repository.

If the script fails at some point, a screenshot (fail.png) and log (fail.log) will be saved. Later, these will be uploaded as assets by GitHub Actions so that we can learn why a pipeline failed.

Just save the following code as run-tests.sh in the root directory of your extension repository. I use a bash script here, however you could also use any other scripting language. Please study the code carefully; I tried to explain everything with inline comments.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
#!/bin/bash

# The script supports two arguments:
#
# -v fedora_version: This determines the version of GNOME Shell to test against.
#                    -v 32: GNOME Shell 3.36
#                    -v 33: GNOME Shell 3.38
#                    -v 34: GNOME Shell 40
#                    -v 35: GNOME Shell 41
#                    -v 36: GNOME Shell 42
# -s session:        This can either be "gnome-xsession" or "gnome-wayland-nested".

# Exit on error.
set -e

usage() {
  echo "Usage: $0 -v fedora_version -s session" >&2
}

FEDORA_VERSION=33
SESSION="gnome-xsession"

while getopts "v:s:h" opt; do
  case $opt in
    v) FEDORA_VERSION="${OPTARG}";;
    s) SESSION="${OPTARG}";;
    h) usage; exit 0;;
    *) usage; exit 1;;
  esac
done

# Go to the repo root.
cd "$( cd "$( dirname "$0" )" && pwd )" || \
  { echo "ERROR: Could not find the repo root."; exit 1; }

IMAGE="ghcr.io/schneegans/gnome-shell-pod-${FEDORA_VERSION}"
EXTENSION="my-cool-extension@my.cool.domain.com"

# Run the container. For more info, visit https://github.com/Schneegans/gnome-shell-pod.
POD=$(podman run --rm --cap-add=SYS_NICE --cap-add=IPC_LOCK -td "${IMAGE}")

# Properly shutdown podman when this script is exited.
quit() {
  podman kill "${POD}"
  wait
}

trap quit INT TERM EXIT

# -------------------------------------------------------------------------------- methods

# This function is used below to execute any shell command inside the running container.
do_in_pod() {
  podman exec --user gnomeshell --workdir /home/gnomeshell "${POD}" set-env.sh "$@"
}

# This is called whenever a test fails. It prints an error message (given as first
# parameter), captures a screenshot to "fail.png" and stores a log in "fail.log".
fail() {
  echo "${1}"
  podman cp "${POD}:/opt/Xvfb_screen0" - | tar xf - --to-command 'convert xwd:- fail.png'
  LOG=$(do_in_pod sudo journalctl)
  echo "${LOG}" > fail.log
  exit 1
}

# This searches the virtual screen of the container for a given target image (first
# parameter). If it is not found, an error message (second parameter) is printed and the
# script exits via the fail() method above.
find_target() {
  echo "Looking for ${1} on the screen."
  POS=$(do_in_pod find-target.sh "${1}") || true
  if [[ -z "${POS}" ]]; then
    fail "${2}"
  fi
}

# This simulates the given keystroke in the container. Simply calling "xdotool key $1"
# sometimes fails to be recognized. Maybe the default 12ms between key-down and key-up
# are too short for xvfb...
send_keystroke() {
  do_in_pod xdotool keydown "${1}"
  sleep 0.5
  do_in_pod xdotool keyup "${1}"
}


# ----------------------------------------------------- wait for the container to start up

echo "Waiting for D-Bus."
do_in_pod wait-user-bus.sh > /dev/null 2>&1


# ----------------------------------------------------- install the to-be-tested extension

echo "Installing extension."

# This directory contains the reference images of the
# settings dialog we will be searching for later.
podman cp "references" "${POD}:/home/gnomeshell/references"

# Copy the extension bundle.
podman cp "${EXTENSION}.zip" "${POD}:/home/gnomeshell"

# Install the extension.
do_in_pod gnome-extensions install "${EXTENSION}.zip"


# ---------------------------------------------------------------------- start GNOME Shell

# Starting with GNOME 40, there is a "Welcome Tour" dialog popping up at first launch.
# We disable this beforehand.
if [[ "${FEDORA_VERSION}" -gt 33 ]]; then
  echo "Disabling welcome tour."
  do_in_pod gsettings set org.gnome.shell welcome-dialog-last-shown-version "999" || true
fi

echo "Starting $(do_in_pod gnome-shell --version)."
do_in_pod systemctl --user start "${SESSION}@:99"
sleep 10

# Enable the extension.
do_in_pod gnome-extensions enable "${EXTENSION}"

# Starting with GNOME 40, the overview is the default mode. We close this here by hitting
# the super key.
if [[ "${FEDORA_VERSION}" -gt 33 ]]; then
  echo "Closing Overview."
  send_keystroke "super"
fi

# Wait until the extension is installed and the overview closed.
sleep 3

# ---------------------------------------------------------------------- perform the tests

# Finally, we open the preferences and check whether the window is shown on screen by
# searching for a small snippet of the preferences dialog.
echo "Opening Preferences."
do_in_pod gnome-extensions prefs "${EXTENSION}"
sleep 3
find_target "references/preferences.png" "Failed to open preferences!"

echo "All tests executed successfully."

The script can be run like this:

make zip                                       # From Part I of the series
./run-tests.sh -v 33 -s gnome-xsession         # Test on Fedora 33 / X11
./run-tests.sh -v 36 -s gnome-wayland-nested   # Test on Fedora 36 / Wayland

This is just a minimal example for how such a testing script could look like. Of course, you can do this completely differently! You could also add a test API to your extension which you call from the script. There are many other things which could be done and with a bit of creativity. For example, the test script of Fly-Pie uses xdotool to move the mouse pointer and to click on menu items.

A more Complex Example: Burn-My-Windows

To test the animations of the Burn-My-Windows extensions, I added a test-mode which can be enabled using gsettings set during the tests. This causes all animations just show one fixed frame for a period of five seconds (this ensures that all screenshots will capture the same moment of the animations). Furthermore, it makes sure that all calls to Math.random() are effectively disabled.

Then, I created a script which generates reference images for all supported GNOME versions / X11 / Wayland / all window-open animations / all window-close animations. This makes up for a total of 136 test cases. Below, you can see the reference images for some included effects. You can observe, how they slightly differ from configuration to configuration.

The test script of Burn-My-Windows then re-captures all those images and compares them with the reference versions.

  Energ. A Energ. B Fire TV Wisps
GNOME 3.36, Wayland, Open
GNOME 3.36, Wayland, Close
GNOME 3.36, X11, Open
GNOME 3.36, X11, Close
GNOME 3.38, Wayland, Open
GNOME 3.38, Wayland, Close
GNOME 3.38, X11, Open
GNOME 3.38, X11, Close
GNOME 40, Wayland, Open
GNOME 40, Wayland, Close
GNOME 40, X11, Open
GNOME 40, X11, Close
GNOME 41, Wayland, Open
GNOME 41, Wayland, Close
GNOME 41, X11, Open
GNOME 41, X11, Close
GNOME 42, Wayland, Open
GNOME 42, Wayland, Close
GNOME 42, X11, Open
GNOME 42, X11, Close

Running the Tests on GitHub

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
name: Tests

on:
  push:
    branches:
      - '**'
  pull_request:
    branches:
      - '**'

jobs:
  tests:
    name: Run Tests
    runs-on: ubuntu-latest
    if: >
      github.event_name == 'pull_request' ||
      ( contains(github.ref, 'main') && !contains(github.event.head_commit.message, '[no-ci]') ) ||
      contains(github.event.head_commit.message, '[run-ci]')
    strategy:
      fail-fast: false
      matrix:
        version:
          - '32'
          - '33'
          - '34'
          - '35'
          - '36'
        session:
          - 'gnome-xsession'
          - 'gnome-wayland-nested'
    steps:
    - uses: actions/checkout@v2
    - name: Download Dependencies
      run: |
        sudo apt update -qq
        sudo apt install gettext -qq
    - name: Build Extension
      run: make zip
    - name: Test Extension
      run: sudo $GITHUB_WORKSPACE/run-test.sh -v ${{ matrix.version }} -s ${{ matrix.session }}
    - uses: actions/upload-artifact@v2
      if: failure()
      with:
        name: log_${{ matrix.version }}_${{ matrix.session }}
        path: fail.log
    - uses: actions/upload-artifact@v2
      if: failure()
      with:
        name: screen_${{ matrix.version }}_${{ matrix.session }}
        path: fail.png

As a final step, we need to run those test via GitHub Actions. To do this, save the above YAML code as .github/workflows/tests.yml in your extension repository. The workflow will be run on each push and each pull request. However, I usually do not need to run all tests on all pushes to branches except for the main branch. Therefore, I added the interesting if in line 15: This ensures that the tests are only executed in three cases:

  • If the push happened to be part of a pull request.
  • If something was pushed to main and the commit message did not contain [no-ci].
  • If something was pushed to any branch and the commit message did contain [run-ci].

The run-tests.sh script will then be executed for each combination of the Fedora versions and Wayland / X11. Whenever a test fails, the fail.png and fail.log will be uploaded. As before, the “Download Dependencies” step may not be necessary for your extension.

Wrapping Up

I hope that you learned something from these guides! Maybe, one or the other aspect can be applied to your extension as well. If you have any questions, suggestions, or alternative solutions, feel free to post a comment!

Comments

blog comments powered by Disqus