Optimizing GitHub Workflows for Efficiency and Sustainability
![Cover image](/optimizing-github-workflows-for-efficiency-and-sustainability/SCWLZmHAc6-300.jpeg)
Original photo by Bruno Figueiredo on Unsplash
Integrating automation into development workflows has become crucial in recent years. In this post, I'll share a few strategies I've found useful for making these workflows both more efficient and sustainable. We'll look into strategies such as canceling redundant jobs, setting appropriate workflow timeouts, optimizing caching techniques, and managing workflow triggers more effectively.
Cancel Redundant Jobs
When active development branches receive rapid updates, whether through direct pushes or pull requests, they often trigger multiple instances of the same workflow. This redundancy can lead to unnecessary resource consumption and potential delays in integration processes.
One effective solution is to utilize concurrency groups. By assigning a concurrency key, you can group workflows, thereby controlling their execution more effectively. GitHub ensures that within a concurrency group, only one job or workflow runs at a time, limiting it to one running and one pending job. Any new jobs that are triggered at that point will instead be queued.
However, by setting cancel-in-progress
to true, we cancel any ongoing workflows within the same concurrency group
when a new one is triggered. This approach effectively clears the queue and ensures that any running workflows are
processing the latest changes:
on:
push:
branches:
- main
concurrency:
group: integration-tests
cancel-in-progress: true
For more granular control, particularly useful in environments that utilize branching strategies, consider using dynamic expressions in the group naming. This configuration allows workflows triggered on different branches to be grouped separately, ensuring that updates on one branch do not interfere with the progress of workflows on another:
concurrency:
group: integration-tests-${{ github.ref }}
cancel-in-progress: true
Employing this strategy optimizes the use of CI resources by avoiding redundant or outdated jobs while maintaining the integrity of our codebase across all types of updates.
Set Workflow Timeouts
The default workflow timeout is 6 hours, which is often excessive for processes that complete within minutes. Setting a more appropriate duration prevents wasteful use of resources, especially in cases where a process might hang due to issues like deadlocks, resource starvation, or software bugs. For instance, a lint check that typically completes within two minutes might only need a 10-minute timeout:
jobs:
lint-checks:
runs-on: ubuntu-latest
timeout-minutes: 10
This simple adjustment prevents resources from being idly consumed, allowing them to be used for more critical tasks and thereby enhancing the sustainability of your workflows.
Utilize Caching for Efficiency
Optimize Dependency Handling
Workflow runs often share the exact same dependencies, which must be downloaded each time. Efficient caching can save both time and resources. I recommend using the setup actions for setting up environments like Node.js with caching enabled:
- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: 20
cache: 'npm'
This configuration caches the ~/.npm
directory, which stores tarballs and metadata rather than the node_modules
,
reducing size and enhancing reusability across different environments.
Using npm ci
instead of npm install
further accelerates builds by installing exact versions from the
package-lock.json
, ensuring reproducible builds and eliminating dependency resolution delays.
Cache and Restore Build Assets
For assets that are resource-intensive to recreate and don't frequently change, consider using the cache action:
jobs:
build:
runs-on: ubuntu-latest
steps:
- name: Cache Assets
uses: actions/cache@v4
with:
path: assets
key: assets-${{ github.run_id }}
restore-keys: |
assets-
This configuration caches the assets
directory, restoring it at the start of each build.
By referencing the run ID in the cache key, each build attempt will initially seek its own unique cache.
If there isn’t an exact match, the restore-keys
option allows the build to use the most recent cache starting
with assets-
. This technique is particularly useful for processes like generating responsive images, where restoring
them from the cache can save significant time and resources.
Note: GitHub retains caches for 7 days after the last access. For infrequent workflows, a storage-based solution might be more appropriate. Cache management, including invalidation, can be handled through the repository's web interface.
Optimize Workflow Triggers
Efficiently using paths
and paths-ignore
to specify which files or directories should trigger workflows can help
focus resources on meaningful changes.
For instance, to exclude documentation updates from triggering workflows:
on:
push:
paths-ignore:
- 'docs/**'
This configuration prevents any change in the docs
directory from triggering the workflow, conserving resources for
non-code updates.
To trigger workflows only for changes in front-end code (e.g., JavaScript files):
on:
push:
paths:
- 'frontend/**.js'
This setting ensures that workflows are triggered only when js
files in the frontend
directory are modified,
ensuring relevant tasks like linting are executed only when necessary.
For more advanced usage, refer to the official GitHub Actions documentation here. Feel free to also check out the paths-filter action which offers more advanced features.
Leverage ARM Architecture for Efficiency
GitHub recently announced ARM-based runners for GitHub Team and Enterprise Cloud plans, with plans to offer them for open-source projects before the year's end.
These runners provide improvements in speed and energy consumption due to their reduced instruction set. They consume 30-40% less energy for some of the most widely deployed workflows and are 37% cheaper than their x64 counterparts.
However, not all workflows are ideal for ARM runners, especially those that depend on software or libraries not yet optimized for ARM architecture, or that rely on x64-specific optimizations. Testing and evaluating your workflows will help you identify any compatibility issues and measure the potential benefits in terms of speed, cost, and energy consumption.
To utilize these ARM runners in your workflows, you first need to create an ARM-based runner as shown in this video
or documented here.
Once set up, you can specify the runner by its name in the runs-on
field in your GitHub Actions workflow. For example:
jobs:
build:
runs-on: my-org-arm-runner
This configuration directs GitHub Actions to use the specified ARM-based runner, allowing you to take advantage of the efficiency and cost savings offered by ARM architecture.
Estimate Workflow Energy Consumption
Determining which strategy consumes fewer resources can sometimes be challenging. Tools like the eco-ci-energy-estimation action allow us to get estimates and help us make informed decisions.
For instance, to determine whether caching npm dependencies would reduce resource consumption, I ran 5 workflow runs with caching and 5 without. The result showed that caching npm dependencies reduced overall energy consumption and time.
The following example demonstrates how to integrate this measurement tool into a workflow. Call start-measurement
before running the work to be measured, and get-measurement
and display-results
after.
- name: Initialize Energy Estimation
uses: green-coding-solutions/eco-ci-energy-estimation@v3
with:
task: start-measurement
- name: Tests measurement
uses: green-coding-solutions/eco-ci-energy-estimation@v3
with:
task: get-measurement
label: 'Install'
- name: Show Energy Results
uses: green-coding-solutions/eco-ci-energy-estimation@v3
with:
task: display-results
Upon completion, the workflow produces a summary that looks like this:
The action sends metrics data to metrics.green-coding.io by default, including energy values, duration of measurements, CPU model, and details about your repository and commit. This data is used to create a badge and a detailed display of your CI runs’ energy consumption, viewable here.
If you prefer not to share this data, simply set send-data
to false
in the action’s configuration.
More Ways to Optimize Workflows
With these simple tips, you can significantly reduce costs, save time, and better allocate resources. For more tips, I highly recommend the presentation "Things your Pipeline Should (Not) Do" by my colleague Raniz, who inspired me to write this post.