Flux Gitops

Introduction to GitOps

Something that is in demand at the time of me writing this post is using gitops in Kubernetes. All over Youtube and other webpages automation is being emphasized. When using Kubernetes there are a few ways to facilitate pushing application updates to a cluster. The method used in this post is flux.

With flux we can make updates to yaml files in code, get a merge request approved by a fellow developer, and then merge. After the merge Flux will take care of the rest. It works by periodically polling your git repo and reconciling any changes made. In this we talk about the basics to start with: bootstrapping, Kustomizations, and HelmReleases.

Prerequisites

kubernetes cluster
flux cli installed

Bootstrapping / Installation

First we need to bootstrap our kubernetes cluster with flux. I use github with deploy keys, but there are many other ways bootstrap a kubernetes cluster. Feel free to bootstrap in a way that meets your needs and go to the next steps.

To bootstrap using Github we follow the instructions (here)[https://fluxcd.io/flux/installation/bootstrap/github/]. First you need a PAT on your Github account. You can find your PATs here. According to the docs when creating a new token you need to ensure it has the following permissions:

Administration -> Access: Read and Write
Contents -> Access: Read and write
Metadata -> Access: Read-only
We are using Read/Write for Administration because flux will be creating its own deploy keys to separate the cluster configuration from our PATs. Save your new key and use it in the next command we run.

Now we bootstrap the cluster using flux.

flux bootstrap github \
    --token-auth=false \ 
    --components-extra=image-reflector-controller,image-automation-controller \
    --owner=youraccountname \
    --repository=yourreponame \
    --branch=main \
    --path=path/in/repo \
    --personal \
    --read-write-key

Let’s break this command down so you can customize as needed.

token-auth=false has the boostrap process create a deploy key in the repo. This means it won’t be setup using a PAT which can expire.
components-extra is specified here because I use the auto image updater in flux. If you don’t know what this is, or have no plan on using it, don’t include this line. You can always rebootstrap with this later if needed with no downsides.
owner is just your username, required.
repository is your repo name, required.
branch is the branch name you want the cluster to watch for changes. I use main, but for a development cluster you can use another one to make it easier to test changes without affecting production.
path is the path in the repo to watch. Some files will be created in the bootstrap process at this location.
personal just means that your account is a personal account and not an org.
read-write-key is used because that allows the deploy key to be read/write to your repository. This isn’t required unless you expect to need your flux installation to update your repo after the bootstrapping process. For me this was used for the auto image updater.

Environment Variables

Make sure to set the following environment variable before bootstrapping:

GITHUB_TOKEN: set to your github PAT

Repository Structure

After bootstrapping the cluster you can start working out your repository. Two examples of this are the official example and my repo. Here is what my repo looks like from the top level:

Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
d-----         3/17/2025   8:40 PM                apps
d-----         3/22/2025   4:34 PM                clusters
d-----          3/5/2025   6:54 AM                infrastructure
-a----         3/22/2025   6:42 PM           1511 README.md

From here on I reference my repository.

clusters: For my path in the bootstrap command I use clusters/clustername. This is where any flux files to deploy apps or infrastructure will be.
apps: Application definition files with the structure apps/app_name.
infrastructure: Much like apps except for backend support like cert-manager or external-dns.
To enable an app for a cluster we use Kustomizations.

Kustomization

The flux Kustomization CRD is used to work like a kustomization.yaml file. It lets flux reconcile code that it points at. An example of a Kustomization looks like this.

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: example
  namespace: flux-system
spec:
  interval: 10m
  targetNamespace: default # optional
  sourceRef:
    kind: GitRepository
    name: flux-system # the repo created by bootstrapping
  path: "../../../apps/website"
  prune: true # if true will delete objects once the flux objects are gone or changed. Otherwise will not delete things.
  timeout: 1m

There are many ways to kustomize the deployments pointed to by these objects. See the docs for the full list of ways to configure these. For example in my repo I like to substitute values from different clusters to reuse code.

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: danstechjourney
  namespace: flux-system
spec:
  interval: 10m
  sourceRef:
    kind: GitRepository
    name: flux-system
  path: "../../../apps/danstechjourney/"
  prune: true
  timeout: 1m
  postBuild:
    substitute:
      domain: danstechjourney.com
      cluster_name: dev
      image: 'registry-dev.martin.lan:443/danstechjourney'
      image_tag: '1742644463'

Using substitution like the above example, I can reuse application code between clusters. If I want to deploy app changes from one cluster to another, I can always use branches to separate and test changes. I add this file under clusters/dev/website.yaml which then connects the cluster to the website code for danstechjourney.

HelmRelease

HelmReleases are used to deploy helm charts via flux. You can specify values for the release as well just like a normal helm chart installation. To use a HelmRelease you first need to define a HelmRepository. For example, this is what I use for deploying vaultwarden.

apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata:
  name: vaultwarden
  namespace: vaultwarden
spec:
  interval: 5m
  url: https://guerzon.github.io/vaultwarden

Now we can deploy the chart via the HelmRelease.

apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: vaultwarden
  namespace: vaultwarden
spec:
  interval: 10m
  timeout: 5m
  chart:
    spec:
      chart: vaultwarden
      version: '0.31.6'
      sourceRef:
        kind: HelmRepository
        name: vaultwarden
      interval: 5m
  releaseName: vaultwarden
  values:
    adminToken:
      existingSecret: "vaultwarden-admin"
      existingSecretKey: "token"
    smtp:
      existingSecret: "vaultwarden-smtp"
      host: smtp.gmail.com
      security: "starttls"
      port: 587
      from: "warden@${domain}"
      fromName: "warden"
      username: 
        existingSecretKey: "username"
      password:
        existingSecretKey: "password"
    storage:
      data:
        name: "vaultwarden-data"
        size: "15Gi"
        path: "/data"
        keepPvc: false
        accessMode: "ReadWriteOnce"
      attachments:
        name: "vaultwarden-files"
        size: "100Gi"
        path: /files
        keepPvc: false
        accessMode: "ReadWriteOnce"
    domain: "https://${domain}"
    invitationOrgName: "${org_name}"
    ingress:
      enabled: true
      class: "nginx"
      nginxIngressAnnotations: true
      additionalAnnotations: 
        external-dns.alpha.kubernetes.io/hostname: "${domain}"
      tls: true
      hostname: "${domain}"
      path: "/"
      pathType: "Prefix"
      tlsSecret: "warden-tls"
      nginxAllowList: ""

You can specify values under spec.values. Under spec.chart the code that identifies the HelmRepository and helm chart/version used.

Deployment

Once the code changes are complete, once you push changes to the target branch your cluster will pickup any differences from what it currently has and start to reconcile the differences. This can take time depending on what you set spec.interval to. One way to force reconcile quickly is to run the flux cli command like so:

flux -n flux-system reconcile kustomization flux-system

This will force a reconcile for flux-system. Once this is done you can force reconcile individual Kustomizations.

flux -n flux-system reconcile kustomization appname

Troubleshooting

If your reconcile takes a while, or if the command to manually reconcile hangs for about a minute, it could be because there is a problem. To look at flux logs you only have to look into the pods that are running in the flux-system namespace.

PS C:\Users\dan\git\danstechjourney> kubectl -n flux-system get pods
NAME                                           READY   STATUS    RESTARTS   AGE
helm-controller-5fc6f89467-8fvdn               1/1     Running   0          17d
image-automation-controller-594bfbc89c-zv5gx   1/1     Running   0          8d
image-reflector-controller-6686fb64c9-z29lt    1/1     Running   0          8d
kustomize-controller-785d866cb7-v4rfr          1/1     Running   0          17d
notification-controller-56776fcb98-phqnm       1/1     Running   0          17d
source-controller-6cd558bc58-pb926             1/1     Running   0          17d

For Kustomization objects we want to look at logs for pods prefixed with kustomize-controller. For HelmRelease objects we want to look at logs from pods prefixed with helm-controller.

Closing

flux is incredible at reconciling differences in code to kubernetes clusters. This post touched upon the basics, but there is so much more you can do with it. Feel free to check out my repo to see what I’m currently using it for. In the future I plan to have a post on auto image updates so be on the lookout for that.

Dan's Tech Journey

Flux Gitops for Kubernetes