Securely Update Kubernetes Secrets with Manual GitHub Workflows

Background

I have my CI/CD pipelines structures so that when I push to the production branch, workflows are triggered to build the image, push the image to the registry, create/update secrets, update the deployment image, etc. However, I recently began to notice this doesn't really align with how I tend to manage secrets in practice.

One issue is that secrets aren't changing nearly as often as code is changing. Generally, these values are only being updated when a third-party service is being included into the project or credentials are being cycled for security purposes. If secrets aren't changing with code, updated secrets on each build is unnecessary.

Next, secrets might be adjusted separately from code being pushed. If I want to reset database credentials but there isn't a software update that coincides, then the workflow isn't of much use to update the credentials. Of course, I could just re-run the last completed workflow, but that will require rebuilding the image unnecessarily and burns actions minutes for work that doesn't need to be performed.

Basically, while managing secrets is an integral process of an CI/CD pipeline, perhaps it shouldn't be so tightly coupled with other steps.

Solution Overview

I determined a good alternative would be to run a separate workflow that's dispatched manually using the workflow_dispatch.

The reason I chose to keep the process to a workflow is that I still want to use GitHub to manage credentials. It's a convenient and secure way to safely manage your secrets, avoid .env files that accidentally find their way into repos, and avoid leakages into console logs when updating secrets from a shell.

To set up the workflow:

• Log into my Kubernetes provider and configure kubectl,
• Delete the existing secret,
• Construct and set the new secret, and
• Rollout a restart of the services that depend on the secrets.

First Things First

Before implementing the above steps, we need to frame out a quick workflow. Workflows should be located relative to your project root and in the directory .github/workflows. To get provide for a basic set up, we'll give this workflow a name, specify the event trigger, provide for a couple environment variables that will help us down the line, and specify the runtime for the runner.

name: Set Secrets

on:
  workflow_dispatch:

env:
  SECRET_NAME: "<secret-name>"
  CLUSTER_NAME: "<cluster-name>"
  NAMESPACE: "default"

jobs:
  set_secrets:
    runs-on: ubuntu-latest
    steps:

Here we're calling the workflow "Set Secrets," we're using the "workflow_dispatch" event trigger which allows us to trigger the workflow manually, and creating the environment variables "SECRET_NAME," "CUSTER_NAME," and "NAMESPACE." Note, namespace only needs to be set if your secrets will live in a namespace other than the default namespace.

Configure kubectl

In order for the workflow to engage with our Kubernetes resources, kubectl needs to be configured to connect with the cluster.

With the cloud provider I use, Digital Ocean (not an endorsement, just who I use), it's as simple as downloading their command line tool (doctl) and passing your access token.

 
      - name: Install doctl
        uses: digitalocean/action-doctl@v2
        with:
          token: ${{ secrets.DIGITALOCEAN_ACCESS_TOKEN }}
      - name: Configure kubectl for DO
        run: |
          set -e
          doctl kubernetes cluster kubeconfig save --expiry-seconds 300 ${{ env.CLUSTER_NAME }}

If you're unfamiliar with how to store secrets in GitHub, see their docs.

Also, I'm running the script with set -e which is an instruction to exit immediately if any command returns a non-zero value indicating an error has likely occurred.

Delete Existing Secret

Now that we're logged in and able to work with our remote resources, we need to delete any existing secrets that would otherwise collide with the secret name specified in the environment variable "SECRET_NAME." If we fail to remove an existing secret of the same name, kubectl create secret will fail.

      - name: Delete existing secret
        run: |
          set -e
          if kubectl get secret ${{ env.SECRET_NAME }} -n ${{ env.NAMESPACE }} >/dev/null 2>&1; then
            kubectl delete secret ${{ env.SECRET_NAME }} -n ${{ env.NAMESPACE }}
          fi

The environment variable "NAMESPACE" is applied to the commands just in case the secret exists in a namespace other than the default one. You only need to set this value if you're using a specific namespace to group your Kubernetes resources.

Last thing worth mentioning is the get secret command writes to /dev/null which is a location where anything written to it is discarded. This is an extra layer of safety since we're dealing with sensitive environment variables.

Construct New Secret

To build the new secret, we're going to create a .env file within the runner that we'll then use in conjunction with the create secret kubectl command. The .env file will be populated using the same convention we used to bring our api access token into this environment.

      - name: Create new secret
        run: |
          set -e
          cat << EOF > .env.prod
            # secrets pulled from gh here
            SECRET_API_KEY=${{ secrets. SECRET_API_KEY }}
            ...
            EOF
            # apply the secret
            kubectl create secret generic ${{ env.SECRET_NAME }} --from-env-file=.env.prod -n ${{ env.NAMESPACE }}
            # delete the .env file (unnecessary because the runner is destroyed after)
            rm -f .env.prod

If you're unfamiliar with the bash cat command used here and without going into too much detail on how this file is being populated and created, the command cat (short for concatenate) is used to read, output and concatenate files. In this situation, the cat command reads the file defined by and contained within the heredoc command (<< EOF ... EOF) and the output is being redirected from the standard output to the .env.prod file using the > redirection operator.

Last, the file is removed with the rm command combined with the -f (force) flag. Is it strictly necessary to remove the .env file? I believe it's good practice to always deliberately clean up sensitive data. However, in the case of GitHub actions runners, they're destroyed afterwards so you're likely safe to omit this.

Restart Relevant Services

So far, we've established a connection with our Kubernetes resources, checked for and deleted any existing secrets, and constructed and set a new secret within our namespace. In order for these secrets to take effect, the services that depend on these services need to be restarted.

Often, the main service that you'll need to restart will be your deployment containing application code. I'll demonstrate this with a restart command for a deployment named 'app.'

      - name: Rollout restart
        run: |
          set -e
          kubectl rollout restart deployment app -n $
          kubectl rollout status deployment app -n $ --timeout=90s

If you have more services that rely on the environment variables set in the secret we've created, add more restarts.

Last, I use rollout status so that way the process watches for the completion of the rollout. This ensures that any downstream workflows in a broader GitHub actions pipeline follow a completed rollout. If you don't have any other workflows, you can omit this line since it will just consume more of your actions minutes unnecessarily.

Complete Solution

 
name: Set Secrets

on:
  workflow_dispatch:

env:
  SECRET_NAME: "<secret-name>"
  CLUSTER_NAME: "<cluster-name>"
  NAMESPACE: "default"

jobs:
  set_secrets:
    runs-on: ubuntu-latest
    steps:
      - name: Install doctl
        uses: digitalocean/action-doctl@v2
        with:
          token: ${{ secrets.DIGITALOCEAN_ACCESS_TOKEN }}
      - name: Configure kubectl for DO
        run: |
          set -e
          doctl kubernetes cluster kubeconfig save --expiry-seconds 300 ${{ env.CLUSTER_NAME }}
      - name: Delete existing secret
        run: |
          set -e
          if kubectl get secret ${{ env.SECRET_NAME }} -n ${{ env.NAMESPACE }} >/dev/null 2>&1; then
            kubectl delete secret ${{ env.SECRET_NAME }} -n ${{ env.NAMESPACE }}
          fi
      - name: Create new secret
        run: |
          set -e
          cat << EOF > .env.prod
            # secrets pulled from gh here
            SECRET_API_KEY=${{ secrets. SECRET_API_KEY }}
            ...
            EOF
            # apply the secret
            kubectl create secret generic ${{ env.SECRET_NAME }} --from-env-file=.env.prod -n ${{ env.NAMESPACE }}
            # delete the .env file (unnecessary because the runner is destroyed after)
            rm -f .env.prod
      - name: Rollout restart
        run: |
          set -e
          kubectl rollout restart deployment app -n ${{ env.NAMESPACE }}
          kubectl rollout status deployment app -n ${{ env.NAMESPACE }} --timeout=90s

Final Thoughts

GitHub actions are a great way to build efficient CI/CD processes. However, from time to time not everything will fit perfectly within a single pipeline. Using the workflow_dispatch event trigger is a great way to handle a workflow that might not need to be executed every time with your primary pipeline or a workflow that needs to be executed without your primary pipeline.

I find this especially true as it relates to maintaining secrets in Kubernetes. I hope what's contained here provides insight on how these tools can be leveraged as well as inspires you to create your own unique and creative workflows.