Getting the Gist of GitHub Actions

Tutorial and tips for GitHub Actions workflows

Table of Contents

• Introduction
• Workflows and actions
  • Getting started
  • Example workflow with one job
  • Example workflow with a build matrix
  • Output
• Pro tips
  • General
  • Secrets
  • Containers
  • VMs and pricing
  • GitHub Actions and Poetry
• Challenges
  • Lacking necessary on: triggers
  • Inability to chain workflows together
  • PR merge conflicts blocking workflow runs
  • Understanding context and expression syntax
• Resources

Introduction

GitHub Actions is a CI/CD service that runs on GitHub repos.

Compared with Travis CI, GitHub Actions is:

• Easier
• More flexible
• More powerful
• More secure

Workflows and actions

Getting started

• Workflows are YAML files stored in the .github/workflows directory of a repository.
• An Action is a package you can import and use in your workflow. GitHub provides an Actions Marketplace to find actions to use in workflows.
• A job is a virtual machine that runs a series of steps. Jobs are parallelized by default, but steps are sequential by default.
• To get started:
  • Navigate to one of your repos
  • Click the "Actions" tab.
  • Select "New workflow"
  • Choose one of the starter workflows. These templates come from actions/starter-workflows.
• Workflows can be triggered by many different events from the GitHub API.
• GitHub provides a context and expression syntax for programmatic control of workflows. For example:

  echo ::set-output name=SPAM_STRING::${{ format(
    'Spam is short for {0} and is made from {1} by {2}',
    'spiced ham',
    'pork shoulder',
    'Hormel'
  ) }}
  

  • Command: echo ::set-output name=ENV_NAME::value, like echo ::set-output name=COLOR::green
  • Expression: ${{ }}
  • Function:
    • contains('this is a demo', 'demo') evaluates to Boolean true
    • format('Spam is short for {0} and is made from {1} by {2}', 'spiced ham', 'pork shoulder', 'Hormel')

Example workflow with one job

A workflow file might look like this:

name: demo

on:
  push:
    branches: [demo]
  pull_request:
  workflow_dispatch:

env:
  APP_NAME: "GitHub Actions demo workflow"

jobs:
  simple:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout repo
        uses: actions/checkout@v2
      - name: Verify the workspace context
        run: echo 'Workspace directory is ${{ github.workspace }}'
      - name: Run a simple echo command with a pre-set environment variable
        run: echo 'Hello World, from ${{ env.APP_NAME }}'
      - name: Set an environment variable using a multi-line string
        run: |
          echo "MULTI_LINE_STRING<<EOF" >> $GITHUB_ENV
          echo "
            Hello World!
            Here's a
            multi-line string.
          " >> $GITHUB_ENV
          echo "EOF" >> $GITHUB_ENV
      - name: Check the environment variable from the previous step
        run: echo $MULTI_LINE_STRING
      - name: Set build environment based on Git branch name
        if: github.ref == 'refs/heads/demo' || contains(env.APP_NAME, 'demo')
        run: echo "BUILD_ENV=demo" >> $GITHUB_ENV
      - name: Use the GitHub Actions format function to provide some details about Spam
        run: |
          echo "SPAM_STRING=${{ format(
            'Spam is short for {0} and is made from {1} by {2}',
            'spiced ham',
            'pork shoulder',
            'Hormel'
          ) }}" >> $GITHUB_ENV
      - name: Run a multi-line shell script block
        run: |
          echo "
          Hello World, from ${{ env.APP_NAME }}!
          Add other actions to build,
          test, and deploy your project.
          "
          if [ "$BUILD_ENV" = "demo" ] || ${{ contains(env.APP_NAME, 'demo') }}; then
            echo "This is a demo."
          elif [ "$BUILD_ENV" ]; then
            echo "BUILD_ENV=$BUILD_ENV"
          else
            echo "There isn't a BUILD_ENV variable set."
          fi
          echo "Did you know that $SPAM_STRING?"
      - uses: actions/setup-python@v2
        with:
          python-version: 3.8
      - name: Run a multi-line Python script block
        shell: python
        run: |
          import os
          import sys

          version = f"{sys.version_info.major}.{sys.version_info.minor}"
          print(f"Hello World, from Python {version} and ${{ env.APP_NAME }}!")
          print(f"Did you know that {os.getenv('SPAM_STRING', 'there is a SPAM_STRING')}?")
      - name: Run an external shell script
        working-directory: ./.github/workflows
        run: . github-actions-workflow-demo.sh
      - name: Run an external Python script
        working-directory: ./.github/workflows
        run: python github-actions-workflow-demo.py

Example workflow with a build matrix

name: demo

on:
  push:
    branches: [master, develop]
  pull_request:
  workflow_dispatch:

env:
  APP_NAME: "GitHub Actions sample workflow with build matrix"

jobs:
  matrix:
    runs-on: ${{ matrix.os }}
    strategy:
      matrix:
        os: [macOS-latest, ubuntu-latest, windows-latest]
        python-version: [3.6, 3.7, 3.8]
        silly-word: [foo, bar, baz]
    steps:
      - uses: actions/checkout@v2
      - name: Echo a silly word
        run: echo ${{ matrix.silly-word }}
      - name: Set up Python ${{ matrix.python-version }}
        uses: actions/setup-python@v2
        with:
          python-version: ${{ matrix.python-version }}
      - name: Run a multi-line Python script block
        shell: python
        run: |
          import os
          import sys

          version = f"{sys.version_info.major}.{sys.version_info.minor}"
          print(f"Hello World, from Python {version}, ${{ matrix.os }}, and ${{ matrix.silly-word }}!")

Output

GitHub Actions provides output like this:

You can see a demo workflow in br3ndonland/algorithms.

Pro tips

General

• Steps in a job are sequential by default.
• Jobs are parallelized by default, unless you control the order by using needs.
• Action inputs and outputs: If you're unclear on what you can do with an action, navigate to the GitHub repo for the action and look for a file called action.yml, like this one in actions/checkout. This file is a manifest declaring what the action can do.
• Debugging: If you want more debugging information, add ACTIONS_STEP_DEBUG to your secrets parameter store.
  • Key: ACTIONS_STEP_DEBUG
  • Value: true

Secrets

Secrets is an encrypted parameter store (key:value store). The syntax is similar to environment variables.

• GitHub Actions can use secrets, so you don't have to hard-code API keys and other credentials. Secrets are redacted from GitHub Actions logs.
• Each repo has a secrets store at Settings -> Secrets (https://github.com/org/repo/settings/secrets)
• Each organization also has a secrets store that can be used in all the organization's repos.

Containers

• Service containers can be set up for test databases and other needs.
• Users can create their own actions. For example, container actions can run in Docker containers of your choosing.

VMs and pricing

• VM info: The GitHub Actions runner provisions virtual machines with similar resources as AWS EC2 t2.large instances.

  • 2-core CPU
  • 7 GB of RAM memory
  • 14 GB of SSD disk space

• GitHub Actions is free for open-source repos. Pricing for other repos only kicks in if you exceed the allotted build minutes.

  

GitHub Actions and Poetry

Poetry is a useful tool for Python packaging and dependency management. The following set of tips was originally posted to python-poetry/poetry#366.

Use caching to speed up workflows

Use actions/cache with a variation on their pip cache example to cache Poetry dependencies for faster installation.

- name: Set up Poetry cache for Python dependencies
  uses: actions/cache@v2
  if: startsWith(runner.os, 'Linux')
  with:
    path: ~/.cache/pypoetry
    key: ${{ runner.os }}-poetry-${{ hashFiles('**/poetry.lock') }}
    restore-keys: ${{ runner.os }}-poetry-

Use the custom installer

Installing Poetry via pip can lead to dependency conflicts, so the custom installer is recommended. The command listed in the docs exits in GitHub Actions with 127 (not on $PATH).

There are some additional modifications needed for GitHub Actions:

• Add -y to avoid prompts.
• Add Poetry to $GITHUB_PATH (note that the ::set-env syntax has been deprecated).
• Move poetry install to separate step to ensure Poetry is on $GITHUB_PATH.

- name: Install Poetry
  run: |
    curl -fsS -o get-poetry.py https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py
    python get-poetry.py -y
    echo "$HOME/.poetry/bin" >> $GITHUB_PATH
- name: Install dependencies
  run: poetry install --no-interaction

Use environment variables for config

Poetry allows config from the poetry config command, or by environment variables. Environment variables are a more dependable way to configure Poetry in CI.

env:
  POETRY_VIRTUALENVS_CREATE: false

Build and publish in one step

• Create a PyPI token.
• Add it to the GitHub Secrets store for the repo (Settings -> Secrets).
• Use the secret in your workflow with ${{ secrets.PYPI_TOKEN }} (secret name is PYPI_TOKEN in this example, and username for PyPI tokens is __token__).
• Use poetry publish --build to build and publish in one step.

- name: Build Python package and publish to PyPI
  if: startsWith(github.ref, 'refs/tags/')
  run: poetry publish --build -u __token__ -p ${{ secrets.PYPI_TOKEN }}

That's why they call it Poetry. Beautiful.

Example workflow

Expand this details element for an example workflow from br3ndonland/inboard that uses these tips.

name: builds

on:
  push:
    branches: [develop, master]
    tags:
      - "[0-9v]+.[0-9]+.[0-9a-z]+"
  workflow_dispatch:

jobs:
  python:
    runs-on: ubuntu-latest
    env:
      POETRY_VIRTUALENVS_CREATE: false
    steps:
      - uses: actions/checkout@v2
      - uses: actions/setup-python@v2
        with:
          python-version: 3.8
      - name: Set up Poetry cache for Python dependencies
        uses: actions/cache@v2
        if: startsWith(runner.os, 'Linux')
        with:
          path: ~/.cache/pypoetry
          key: ${{ runner.os }}-poetry-${{ hashFiles('**/poetry.lock') }}
          restore-keys: ${{ runner.os }}-poetry-
      - name: Set up pre-commit cache
        uses: actions/cache@v2
        if: startsWith(runner.os, 'Linux')
        with:
          path: ~/.cache/pre-commit
          key: ${{ runner.os }}-pre-commit-${{ hashFiles('.pre-commit-config.yaml') }}
          restore-keys: ${{ runner.os }}-pre-commit-
      - name: Install Poetry
        run: |
          curl -fsS -o get-poetry.py https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py
          python get-poetry.py -y
          echo "$HOME/.poetry/bin" >> $GITHUB_PATH
      - name: Install dependencies
        run: poetry install --no-interaction -E fastapi
      - name: Run pre-commit hooks
        run: pre-commit run --all-files
      - name: Run unit tests
        run: pytest
      - name: Build Python package and publish to PyPI
        if: startsWith(github.ref, 'refs/tags/')
        run: poetry publish --build -u __token__ -p ${{ secrets.PYPI_TOKEN }}
  docker:
    runs-on: ubuntu-latest
    needs: [python]
    steps:
      - uses: actions/checkout@v2
      - name: Log in to Docker registry
        run: docker login ghcr.io -u ${{ github.actor }} -p ${{ secrets.PAT_GHCR }}
      - name: Build Docker images
        run: |
          docker build . --rm --target base -t ghcr.io/br3ndonland/inboard:base --cache-from python:3.8
          docker build . --rm --target starlette -t ghcr.io/br3ndonland/inboard:starlette
          docker build . --rm --target fastapi -t ghcr.io/br3ndonland/inboard:fastapi
      - name: Push Docker images to registry
        run: |
          docker push ghcr.io/br3ndonland/inboard:base
          docker push ghcr.io/br3ndonland/inboard:starlette
          docker push ghcr.io/br3ndonland/inboard:fastapi
      - name: Add Git tag to Docker images
        if: startsWith(github.ref, 'refs/tags/')
        run: |
          GIT_TAG=$(echo ${{ github.ref }} | cut -d / -f 3)
          docker tag ghcr.io/br3ndonland/inboard:base ghcr.io/br3ndonland/inboard:base-"$GIT_TAG"
          docker tag ghcr.io/br3ndonland/inboard:starlette ghcr.io/br3ndonland/inboard:starlette-"$GIT_TAG"
          docker tag ghcr.io/br3ndonland/inboard:fastapi ghcr.io/br3ndonland/inboard:fastapi-"$GIT_TAG"
          docker push ghcr.io/br3ndonland/inboard:base-"$GIT_TAG"
          docker push ghcr.io/br3ndonland/inboard:starlette-"$GIT_TAG"
          docker push ghcr.io/br3ndonland/inboard:fastapi-"$GIT_TAG"
      - name: Tag and push latest image
        run: |
          docker tag ghcr.io/br3ndonland/inboard:fastapi ghcr.io/br3ndonland/inboard:latest
          docker push ghcr.io/br3ndonland/inboard:latest

Bonus: automated dependency updates with Dependabot

Dependabot now offers automated version updates, with (preliminary) support for Poetry 🎉. If you have access to the Dependabot beta, set up .github/dependabot.yml as described in the docs:

version: 2
updates:
  - package-ecosystem: "github-actions"
    directory: "/"
    schedule:
      interval: "weekly"
  - package-ecosystem: "pip"
    directory: "/"
    schedule:
      interval: "weekly"

Dependabot will now send you PRs when dependency updates are available. Although package-ecosystem must be set to pip, it will pick up the pyproject.toml and poetry.lock. Check the status of the repo at Insights -> Dependency graph -> Dependabot.

Challenges

Lacking necessary on: triggers

There's no PR merge trigger. There are two ways to implement this:

1. Use pull_request: types: [closed] as the event trigger and only run each job if the event payload contains merged == 'true'. See the context and expression syntax docs for examples of how to use the event payload.

   name: demo
   on:
     pull_request:
       types: [closed]
     # release:
     #   types: [published]
   jobs:
     job1:
     # if: github.event.pull_request.merged == 'true' || github.event.release.draft == 'false'
     if: github.event.pull_request.merged == 'true'
     steps:
       - name: step1
         run: echo 'hello world'

2. Set up a repository_dispatch or workflow_dispatch webhook and use that as the trigger. The method for this is unclear, and workflow_dispatch events may only be read on the default branch.

Inability to set environment variables per-trigger: It's difficult to set environment variables per-trigger, such as based on which branch was checked out. There's a top-level env: key, but it doesn't allow expressions or separate steps:.

env: # Can't do this
  - name: Set build environment to production if master branch is checked out
    if: contains(github.ref, 'master')
    run: echo "BUILD_ENV=production" >> $GITHUB_ENV
  - name: Set build environment to development if develop branch is checked out
    if: contains(github.ref, 'develop')
    run: echo "BUILD_ENV=development" >> $GITHUB_ENV
  - name: Set build environment to test otherwise
    if: ${{ !contains(github.ref, 'master') || !contains(github.ref, 'develop') }}
    run: echo "BUILD_ENV=test" >> $GITHUB_ENV

Furthermore, environment variables set to $GITHUB_ENV within a job are scoped to that job.

The solution is to use outputs instead of environment variables. Set outputs from one job and read them in downstream jobs. Note that step outputs must also be set as job outputs in order to be passed to other workflows.

Inability to chain workflows together

• The needs: feature is great, but I should be able to depend on other workflows, rather than having all the workflows in one file.
• workflow_dispatch: seems like it could be what I'm looking for, but it's marketed as a way to manually trigger workflows in the UI, not as a way to automatically chain workflows together.
• There's a new workflow_run trigger that may help to chain workflows together.
  • GitHub blog
  • GitHub community
  • GitHub Actions docs
  • GitHub REST API docs

PR merge conflicts blocking workflow runs

• If a PR has merge conflicts, GitHub Actions workflows may not run at all. See this GitHub community thread.

• Try using equality operators (== and !=) to check out the PR HEAD commit, instead of the default (the merge commit), as described in the actions/checkout README:

  - uses: actions/checkout@v2
    if: ${{ github.event_name != 'pull_request' }}
  - uses: actions/checkout@v2
    if: ${{ github.event_name == 'pull_request' }}
    with:
      ref: ${{ github.event.pull_request.head.sha }}

• If checking out the HEAD commit doesn't work, you may need to resolve merge conflicts to continue.

Understanding context and expression syntax

• See the context and expression syntax docs

• In which context can I use which things?

  • Where can I define env:?
  • Where can I define defaults:?
  • Where can I add if:?
  • When do I need ${{ }} for expressions? The docs say, "When you use expressions in an if conditional, you may omit the expression syntax (${{ }}) because GitHub automatically evaluates the if conditional as an expression, unless the expression contains any operators. If the expression contains any operators, the expression must be contained within ${{ }} to explicitly mark it for evaluation." However, if: conditionals with expressions seem to work fine without ${{ }}.

• Object filters seem like a useful corollary to CloudFormation mappings, but there's no explanation for how to set up object filters within workflows.

• There's no concept of if/elif/else.

• Syntactic subtleties, such as the requirement for single quotes in some YAML fields:

  jobs:
    base:
      runs-on: ubuntu-latest
      steps:
        # this works
        - name: Set build environment to production
          if: github.ref == 'refs/heads/master'
          run: echo "BUILD_ENV=production" >> $GITHUB_ENV
        # this doesn't
        - name: Set build environment to production
          if: github.ref == "refs/heads/master"
          run: echo "BUILD_ENV=production" >> $GITHUB_ENV

• github.ref vs github.base_ref: Confusingly, github.base_ref appears to output different syntax than github.ref.

  • github.base_ref returns just the branch name, like develop.
  • github.ref returns the full Git ref, like refs/heads/develop.

Resources

• Overview
  • GitHub Actions Cheat Sheet
  • GitHub docs: Actions - Core concepts
  • GitHub docs: Actions - Configuring and managing workflows
• Workflow triggers (the on: section of YAML workflow files):
  • GitHub docs: Actions - Events that trigger workflows
  • GitHub Blog 20200706: GitHub Actions - Manual triggers with workflow_dispatch
• Workflow syntax
  • GitHub docs: Actions - Workflow syntax reference
  • GitHub docs: Actions - Context and expression syntax (syntax in double curly braces like ${{ github.repository }}, and functions like format())
  • GitHub docs: Actions - Workflow commands for GitHub Actions (shell commands you can perform within workflows, like echo ::set-env name=COLOR::green)
• GitHub Actions + GitHub Container Registry
  • GitHub Blog 20200901: Introducing GitHub Container Registry
  • GitHub docs: Getting started with GitHub Container Registry
• Actions
  • GitHub Actions Marketplace
  • actions/runner: the runner for GitHub Actions
  • sdras/awesome-actions

GitHub Gist notes

• A Gist is actually a repository.
• To clone the Gist locally:

  git clone git@gist.github.com:f9c753eb27381f97336aa21b8d932be6.git github-actions

• No subdirectories are allowed.
• To add images to a Markdown file in a Gist:
  • Commit the image file (in the same directory, no sub-directories)
  • Push the change to GitHub with git push
  • Click on the "raw" button next to the image
  • Copy the URL
  • Add the URL to an image tag in the Markdown file.