I'm using GitHub Actions a lot at work. I think it's simple and effective enough to be understood by anybody and is a great product when it comes to start playing with continuous integration. In my recent work, I've been managing some company self-hosted runners so that we could interact with some on premise APIs. However, there's something that still questions me : how such a system works ? How, as a simple developer, all you need to focus on is a simple yml file in your .github/workflows directory ? Let's try to figure it out and let's implement our own GitHub actions clone.

This article is the first one of a series. I don't know how many articles I'll wrote about, but it seems that building a CI solution is not something you would resume in a single blog post. My guess on that is that I'll probably write two, three articles on the specs (basically understanding how GitHub actions works, how workflows & jobs are scheduled and so on), then I'll get some fun about CODING and then I'll write some feedback & critics about the implementation.

Well, there's a lot of work ahead ! Let's go ! 💪

What are we building here ?

The Github Actions spec

My goal is not to recreate a full CI application. In this post series I'll just spec & implement an app that is able to parse a simplified GitHub actions workflow spec. As a reference, here's the workflow that I'd like to implement (considering that every property which is not in that example won't be implemented) :

name: My Workflow

on:
  workflow_dispatch:
  inputs:
      environment_name:
        description: 'Environment to run tests against'
        type: string
        required: true
  # push:
  # schedule:
  #   - cron:  '30 5,17 * * *'

jobs:
  notify_start:
    name: notify start
    runs-on:
      - tag1
    steps:
      - name: notify
        run: echo "Hey, workflow started !"
  test_env:
    name: test app
    needs:
      - notify_start
    runs-on:
      - tag1
      - tag2
    steps:
      - name: build the app
        run: |
          echo "Building the app..."
          sleep 20
          echo "done"
      - name: test the app
        run: |
          echo "Let's test the app !"
          sleep 10
          echo "end of test against ${{ inputs.environment_name }}"
  test_security:
    name: test app security
    needs:
      - notify_start
    runs-on:
      - tag1
      - tag2
    steps:
      - name: test the app security
        run: |
          echo "Let's test the security of the app !"
          sleep 40
          echo "end of test against ${{ inputs.environment_name }}"
  notify_end:
    name: notify end
    needs:
      - test_app
      - test_security
    runs-on:
      - tag1
    steps:
      - name: notify end
        run: echo "Hey, workflow ended !"

For now, I won't focus on the uses syntax as I'm more interested about the workflow process than the job execution and actions support. I also won't support jobs running in containers using the container syntax.

Triggering and preview/mockups

I'd like to provide a web UI where the user could upload such a file through a form and then list its workflows. Each workflow will be runnable using a form that will let the user simulate the trigger type if needed.

You must also be able to get the current workflow run history :

For every workflow I want to have a graphical visualization of the execution:

And be able to read the execution logs for each job step:

The components

Here are the main components that I already discovered as mandatory in such an app :

The workflow parser

There are many components to design. The most obvious part is that we'll need a github actions spec parser, that will allow us to read a yaml file, inspect it and trigger some background actions if required. The parser will be used everytime a user upload a new workflow. If the spec is not valid, the workflow will be rejected. Otherwise, the app will register the workflow "hooks", ie when the workflow should run (in our case, we'll only cover the schedule event, but it should also be any push, pull_request event and so on.

The workflow scheduler

That component is in charge of scheduling a workflow execution. Every job defined in a workflow must run on a runner. These are defined in the runs-on section. In our case, the runner will be some kind of message queue consumers. If there's no runner matching a job requirements (based on the job tags defined in the runs-on property, then the job will not be able to start. The scheduler is also in charge of updating the workflow run status when it receives some updates (like marking the workflow as success/failure).

The runners

Those are in charge of the jobs executions. Our runners will be some background processes that we'll be able to launch with some tags, so that we'll be able to mock the ubuntu-latest, self-hosted foo-bar tags that GitHub Actions implements.

The UI

To follow all the background work, The UI we sketched before will be very important. It must follow every workflow update. I'd like the UI to be updated in real time when something happens.

Getting to work

My first assumptions about the app is to start designing a good spec parser. That parser will be used when a workflow is created/updated/deleted from the UI, and will register/unregister some events in the workflow scheduler.

When those events are triggered (from the UI or a cron schedule), the workflow scheduler will evaluate the corresponding workflow and try to find any available runner able to run the jobs that are immediately runnable.

When a runner starts and ends a job, it will persist the job state in a database and notify the workflow scheduler so that it will reevaluate the jobs to run until all the workflow jobs are executed or one of them failed.

I think we got some stuff to get started ! And that's gonna be a lot of work !