Terraform pull request automation that runs inside of Kubernetes

github.com/runatlantis/atlantis

Updated: 2023-08-21

Additional documentation can be found on there website.

In my setup I will use a GitHub app to integrate with Atlantis

Where your see the following replace with your own

Key Description Example
ORG Your GitHub organization or GitHub username github
URL Your domain name infrasec.sh

Initial Setup

The first step is to bootstrap the GitHub app.

Pod Bootstrap

  • ATLANTIS_ATLANTIS_URL is the endpoint that will be exposed to the internet. I recommend to whitelist or user rules (traefik) to prevent anyone from accessing all endpoints.
  • ATLANTIS_GH_ORG & ATLANTIS_REPO_ALLOWLIST need your username or organizations name.
  • The rest can be left as defaults
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

apiVersion: v1
kind: Pod
metadata:
  name: atlantis-bootstrap
  namespace: atlantis
spec:
  containers:
    - name: atlantis
      image: ghcr.io/runatlantis/atlantis:v0.22.3-alpine
      env:
        - name: ATLANTIS_ATLANTIS_URL
          value: https://atlantis.svc.infrasec.sh
        - name: ATLANTIS_GH_ORG
          value: github
        - name: ATLANTIS_REPO_ALLOWLIST
          value: github.com/github/*
        - name: ATLANTIS_PORT
          value: "4141"
        - name: ATLANTIS_GH_USER
          value: fake
        - name: ATLANTIS_GH_TOKEN
          value: fake
      ports:
        - name: atlantis
          containerPort: 4141

Next you need to go to the atlantis url or port forward to setup the GitHub app.

https://atlantis.svc.infrasec.sh/github-app/setup

Atlantis create github app

Click setup and follow through the GitHub steps for setting up an app.

Make sure to save the gh-app-id, gh-app-key-file, and gh-webhook-secret.

github app credentials

Follow the link to add the app. Select the repositories you would like to allow it access to.

You can now delete the pod used to bootstrap the app.

Setup Secrets

Create a new kubernetes secret (atlantis-secret) in the same namespace (atlantis) with the following values we got from setting up the GitHub app.

Key Value
ATLANTIS_GH_APP_ID gh-app-id
ATLANTIS_GH_WEBHOOK_SECRET gh-webhook-secret
ATLANTIS_GH_APP_KEY gh-app-key-file

Installation

Configurations

Create a new config map, it will be used to store custom configuration that Atlantis will use.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

apiVersion: v1
kind: ConfigMap
metadata:
  name: atlantis-cm
  namespace: atlantis
data:
  server-config.yaml: |
    metrics:
      prometheus:
        endpoint: "/metrics"
  repo-config.yaml: |
    repos:
    - id: /.*/
      apply_requirements: [approved]
      allowed_workflows: [custom]
      allowed_overrides: [workflow]
      workflow: custom
    workflows:
      custom:
        plan:
          steps:
          - init
          - plan
        apply:
          steps:
          - apply
  • The server config will only going to setup a prometheus metrics endpoint.
  • The repository config will be applied to all projects and will run a default workflow of init, plan, and apply.

Application

Now we will create the container that will run Atlantis, its a StatefulSet because it needs a persistent incase the pod restarts.

Update the environment variable ATLANTIS_ATLANTIS_URL to your domain name.

What else does it do:

  • The secret we created earlier is pulled into the containers environment variables
  • Volumes are mounted containing our configmap and secret GitHub key for Atlantis
  • We set resource requirements, my guess of what it they should be
  • Create a five Gi volume that is used to store persistent plan data between pods
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: atlantis
  namespace: atlantis
spec:
  serviceName: atlantis
  replicas: 1
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      partition: 0
  selector:
    matchLabels:
      app: atlantis
  template:
    metadata:
      labels:
        app: atlantis
    spec:
      securityContext:
        fsGroup: 1000
      containers:
        - name: atlantis
          image: ghcr.io/runatlantis/atlantis:v0.22.3-alpine
          env:
            - name: ATLANTIS_WRITE_GIT_CREDS
              value: "true"
            - name: ATLANTIS_GH_APP_KEY_FILE
              value: /etc/secrets/ATLANTIS_GH_APP_KEY
            - name: ATLANTIS_CONFIG
              value: /etc/config/server-config.yaml
            - name: ATLANTIS_REPO_CONFIG
              value: /etc/config/repo-config.yaml
            - name: ATLANTIS_DATA_DIR
              value: /atlantis
            - name: ATLANTIS_PORT
              value: "4141"
            - name: ATLANTIS_ATLANTIS_URL
              value: https://atlantis.svc.infrasec.sh
            - name: ATLANTIS_REPO_ALLOWLIST
              value: "*"
            - name: ATLANTIS_GH_APP_ID
              valueFrom:
                secretKeyRef:
                  name: atlantis-secret
                  key: ATLANTIS_GH_APP_ID
            - name: ATLANTIS_GH_WEBHOOK_SECRET
              valueFrom:
                secretKeyRef:
                  name: atlantis-secret
                  key: ATLANTIS_GH_WEBHOOK_SECRET
          volumeMounts:
            - name: atlantis-data
              mountPath: /atlantis
            - name: key-mount
              mountPath: "/etc/secrets"
              readOnly: true
            - name: config
              mountPath: "/etc/config"
          ports:
            - name: atlantis
              containerPort: 4141
          resources:
            requests:
              memory: 256Mi
              cpu: 250m
            limits:
              memory: 512Mi
              cpu: 500m
          livenessProbe:
            periodSeconds: 60
            httpGet:
              path: /healthz
              port: 4141
              scheme: HTTP
          readinessProbe:
            periodSeconds: 60
            httpGet:
              path: /healthz
              port: 4141
              scheme: HTTP
      volumes:
        - name: key-mount
          secret:
            secretName: atlantis-secret
            items:
              - key: ATLANTIS_GH_APP_KEY
                path: ATLANTIS_GH_APP_KEY
        - name: config
          configMap:
            name: atlantis-cm
  volumeClaimTemplates:
    - metadata:
        name: atlantis-data
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 5Gi

Depending on your workload you may need additional access to run your Terraform code. This can be done by adding it as environment variables or if you need AWS use service account roles.

Usage

Now when you open a pull request in an allowed repository, Atlantis will automatically run a plan and show you the changes that will be applied.

atlantis plan in a github pull request

Once your ready to apply comment atlantis apply

Notes

If you need custom configuration for a single project add to the root of the repo a file called atlantis.yaml. See documentation on configuring the file here

With having something automatically run code security issues arise, such as grabbing credentials.

There is no perfect solution to this I have found, but here are some of my thoughts on it

  • Script to check providers being used vs a whitelist
  • Scan code to look for provisioners (local & remote exec)

Atlantis has built in OPA checks using conftest.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

package custom.aws.provisioners.local_exec

local_exec_provisioners = all {
	all := [name |
		name := input.configuration.root_module.resources[_]
		name.provisioners[_].type == "local-exec"
	]
}

deny[msg] {
	local_exec_provisioners > 0
	msg := "local-exec provisioners cannot be used"
}

Whats Next

Create custom workflows that will lint your code and check it for security issues.

Send webhooks when Atlantis deploys your code

If there are any questions reach out to me or on Atlantis Slack.