I’ve been running Kaniko for several years without any major issues. Recently, when I was updating and applying patches to my CI systems on Argo Workflows, I discovered that the Kaniko project is now (June 3, 2025) archived.

It can now be added to Google Graveyard

Kaniko

Kaniko is nice since it’s super simple to deploy and get working.

docker run --rm -v $(pwd):/workspace \
  gcr.io/kaniko-project/executor:latest \
  --dockerfile=Dockerfile --context=/workspace \
  --no-push --reproducible --skip-unused-stages=true

The major downside I’ve run into is getting secrets into the container.

I’ve found two workarounds (helpful blog):

  1. Copy the short-lived secret in with the Dockerfile, with the hope that it doesn’t get copied into the final image.
  2. Mount the secret into /kaniko/SECRET_NAME in the kaniko container, this /kaniko path is available from inside the image during runtime so it can be read from there.

BuildKit

I settled on switching to using BuildKit as Kaniko’s replacement.

It requires a little more setup but covers all of Kaniko’s features with the following upsides:

  • Faster builds
  • Slightly smaller images (in KB)
  • Fully secret support
  • Easier to run locally (available directly in docker)
docker run --rm --privileged --entrypoint buildctl-daemonless.sh \
  -v $HOME/.npmrc:/secrets/npmrc:ro -v $(PWD):/workspace \
  -e BUILDKITD_FLAGS='--oci-worker-no-process-sandbox' \
  moby/buildkit:rootless \
  build --frontend dockerfile.v0 --local context='/workspace' --local dockerfile='/workspace' \
  --secret id=npmrc,src=/secrets/npmrc \
  --output type=image,name=ttl.sh/buildkit-demo:dummy,push=true

Pushing to ECR

The default BuildKit image cannot automatically authenticate to an AWS ECR repository.

To work around that the following Dockerfile can be used to install the “amazon-ecr-credential-helper” and setup the config.

1
2
3
4
5
6
7

FROM moby/buildkit:rootless

RUN mkdir -p ~/.docker && \
    echo "{\"credsStore\":\"ecr-login\"}" > ~/.docker/config.json

# https://github.com/awslabs/amazon-ecr-credential-helper/releases
ADD --chown=root:user --chmod=050 https://amazon-ecr-credential-helper-releases.s3.us-east-2.amazonaws.com/0.10.1/linux-amd64/docker-credential-ecr-login /usr/local/bin/docker-credential-ecr-login

If you’re running locally, you’ll need to mount your local AWS credentials into the container.

-v $HOME/.aws:/home/user/.aws:ro -e AWS_PROFILE=ecr-dummy-profile

This setup was inspired by the setup from this blog

Misc

⚠️

If your trying to push to multiple registries with output type image and oci/docker the image digests will be different.

My solution for getting the hashes to be the same is just export as an oci tar and then using crane in both environments to upload the images to the registries. I’ll upload the fixed code to GitHub when I get around to the full argo workflows build-out.

If you need the container digest (hash) the following argument can be added. During the build a json object containing image hash and other information is written to that path.

--metadata-file /workspace/metadata.json

Multiple outputs can be setup (remote or local) in my case to support multi-repos I use Crane which requires the image archive.

--output type=oci,dest=/workspace/image.tar

Improvements

My setup could be improved, for one, it does not support any types of caching. A dedicated build node could be set up for image caching or a cache image could be pulled to reuse layers. (sparkfabrik blog)

Argo Workflows

BuildKit is pretty much a drop in replacement in Argo Workflows with only needing a few changes.

  1. Kubernetes Nodes must support user namespaces if running non-privileged (bottlerocket)
  2. Image digest must be parsed. {{=jsonpath(tasks.buildkit.outputs.parameters.digest, ''$.["containerimage.digest"]'')}}

Here is an example Helm WorkflowTemplate that builds images. I’ll eventually have a full Argo Workflows CI system document but for now this will make due.

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164

---
apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
  name: build-nodejs
  namespace: {{ .Values.namespace }}
spec:
  arguments:
    parameters:
      - name: build_path
      - name: commit
      - name: destination_image
      - name: ecr_repository
      - name: project
      - name: release_stage
      - name: source_workflow
      - name: stack
  entrypoint: entrypoint
  serviceAccountName: argo-nodejs-build
  podGC:
    strategy: OnWorkflowCompletion
    deleteDelayDuration: 600s
  templates:
    - name: entrypoint
      inputs:
        parameters:
          - name: build_path
          - name: commit
          - name: destination_image
          - name: ecr_repository
          - name: project
          - name: release_stage
          - name: stack
      dag:
        tasks:
          - name: code-artifact-auth
            template: code-artifact-auth
          - name: buildkit
            depends: code-artifact-auth
            template: buildkit
            arguments:
              parameters:
                - name: build_path
                  value: "{{ `{{ inputs.parameters.build_path }}` }}"
                - name: destination_image
                  value: "{{ `{{ inputs.parameters.destination_image }}` }}"
          - name: notification
            depends: buildkit
            templateRef:
              name: success-notification
              template: post
            arguments:
              parameters:
                - name: commit
                  value: "{{ `{{ inputs.parameters.commit }}` }}"
                - name: releaseStage
                  value: "{{ `{{ inputs.parameters.release_stage }}` }}"
                - name: repo
                  value: "{{ `{{ inputs.parameters.stack }}` }}/{{ `{{ inputs.parameters.project }}` }}"
                - name: webhook_secret
                  value: success-notification
          - name: commit-changes
            depends: buildkit.Succeeded
            templateRef:
              name: git-committer
              template: commit-image
            arguments:
              parameters:
                - name: release_stage
                  value: "{{ `{{ inputs.parameters.release_stage }}` }}"
                - name: hash
                  value: '{{ `{{=jsonpath(tasks.buildkit.outputs.parameters.digest, ''$.["containerimage.digest"]'')}}` }}'
                - name: project
                  value: "{{ `{{ inputs.parameters.project }}` }}"
                - name: image_tag
                  value: "{{ `{{ inputs.parameters.commit }}` }}"
                - name: stack
                  value: "{{ `{{ inputs.parameters.stack }}` }}"
    - name: code-artifact-auth
      podSpecPatch: '{"serviceAccountName": "argo-nodejs-build"}'
      container:
        image: public.ecr.aws/aws-cli/aws-cli:2.31.18@sha256:d3f5b7078a71fd7cb488719571c50abd252f4320cdf375601f2d6d40caf97b55
        command: ["sh", "-c"]
        args: [
          'echo "@example:registry=https://example-0123456789.d.codeartifact.us-east-1.amazonaws.com/npm/npm/" > /source/.npmrc &&
          echo "//example-0123456789.d.codeartifact.us-east-1.amazonaws.com/npm/npm/:_authToken=$(aws codeartifact get-authorization-token --domain example --query authorizationToken  --duration-seconds 900 --output text)"  >> /source/.npmrc'
        ]
        resources:
          limits:
            cpu: 25m
            memory: 100Mi
          requests:
            cpu: 5m
            memory: 5Mi
        volumeMounts:
          - mountPath: /source
            name: source-code
    - name: buildkit
      podSpecPatch: '{"serviceAccountName": "argo-nodejs-build"}'
      inputs:
        parameters:
          - name: build_path
          - name: destination_image
      container:
        command:
          - buildctl-daemonless.sh
        workingDir: "/source/{{ `{{ inputs.parameters.build_path }}` }}"
        args:
          - --log-format
          - json
          - build
          - --frontend
          - dockerfile.v0
          - --local
          - context=.
          - --local
          - dockerfile=.
          - --secret
          - id=npmrc,src=/buildkit/secrets/.npmrc
          - --output
          - type=image,name={{ `{{ inputs.parameters.destination_image }}` }},push=true
          - --output
          - type=docker,dest=/source/image.tar
          - --metadata-file
          - /tmp/digest
        securityContext:
          seccompProfile:
            type: Unconfined
          appArmorProfile:
            type: Unconfined
          runAsUser: 1000
          runAsGroup: 1000
        image: 0123456789.dkr.ecr.us-east-1.amazonaws.com/buildkit:dummy@sha256:abc123
        env:
          - name: BUILDKITD_FLAGS
            value: --oci-worker-no-process-sandbox
        resources:
          limits:
            cpu: 200m
          memory: 1Gi
          requests:
            cpu: 50m
            memory: 100Mi
        volumeMounts:
          - mountPath: /source
            name: source-code
          - mountPath: /buildkit/secrets/.npmrc
            name: source-code
            readOnly: true
            subPath: .npmrc
      outputs:
        parameters:
          - name: digest
            valueFrom:
              default: "{}"
              path: /tmp/digest
  volumes:
    - name: source-code
      persistentVolumeClaim:
        claimName: "{{ `{{ workflow.parameters.source_workflow }}` }}-source-code"
    - name: git-secret
      secret:
        defaultMode: 256
        secretName: argo-workflows-ssh