Almost exactly three years ago I wrote a blog on my personal page entitled Testing pgAdmin which went into great detail discussing how we test pgAdmin prior to releases. Back then, all of the automated testing was performed using Jenkins, with a number of jobs that ran various test suites whenever new code was checked in. All of this infrastructure ran in a virtual private cloud on AWS consisting of a large number of virtual machines and other resources, hosted for the project by EDB.
I recently undertook a project to move the testing infrastructure to our Github project, using Github Actions. This gives us a number of benefits:
- Actions can be set to run against pull requests as well as commits. This allows us to commit changes safe in the knowledge that they will not break the build.
- pgAdmin developers outside of the primary team at EDB can run the tests easily in their own forks of the Github repo, without needing access to our secure build/test infrastructure.
- The tests are defined as part of the source code in the pgAdmin repository, rather than as a subset of a number of other Jenkins jobs.
- Costs are reduced as Github offer free use of Action "runners" to open source projects, so we no longer need to maintain such a large build/test environment.
Moving the tests across to Github actions was relatively simple, but did take a fair amount of experimentation and learning. Each of the major tests (or test suites) we run is defined as a workflow in the source tree - each one is a YAML file that defines a series of actions to take, and when to take them. Here's a simple example from check-container-build.yml:
name: Check container build
on:
push:
branches: [ "master" ]
pull_request:
branches: [ "master" ]
workflow_dispatch:
concurrency:
group: '${{ github.workflow }} @ ${{ github.event.pull_request.head.label || github.head_ref || github.ref }}'
cancel-in-progress: true
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Build the container
run: docker build .
The name value is obvious, so I won't bother to describe it!
The on section is used to tell Github Actions when the action should be run; in this case, on a commit or pull request against the master branch, or if manually triggered on the website (workflow_dispatch).
The concurrency section is used to define what happens if multiple workflows get triggered at once. The group lets us define how we recognise other instances of the same workflow, and cancel-in-progress is used to cancel an existing instance of a workflow in the same group (i.e. the same workflow on the same branch, as we have it here). There is no point in wasting compute resources on completing a workflow, if another commit is made halfway through the run.
The jobs section is where things get more interesting. We specify that we want to run this step on an ubuntu-latest runner, and then we use an action to checkout the code. Finally, we attempt to build the container.
Github provides a nice user interface for viewing the test runs that have executed; note that not all of the information is visible to users that aren't part of the project on Github, but it would be on your own forks of the repository.
A number of our workflows are more complex than the simple example above. In order to run tests against PostgreSQL and EDB Advanced Server we run "matrix" actions in which the tests run on each combination of database server version and platform. This leads to very large numbers of tests being run in total across those combinations, but with quite simple code to make that happen - most of the complexity comes from having to make the steps work as needed on each different platform.
One final point to note is that a small number of the workflows require access to third party resources that some developers may not have; for example, the EDB Advanced Server package repositories, or a SonarQube server for code analysis. In these cases, those tests are not run unless variables and secrets are configured in your Github account to provide the required information to access those resources.
Moving the automated testing of pgAdmin to Github has helped the project improve automated testing; most importantly by co-locating the tests with the code, and allowing them to be run against pull requests. In the future – once we're happy the tests are stable – we'll be able to further tighten up the development process by setting rules that require all tests to be green on a pull request before it can be merged for example.