GitOps: The Good, The Bad, and The Repo

Balhau

— 5 min read

How one Ansible playbook bootstrapped a Kubernetes cluster into a GitOps workflow — and why you might want to do the same (or not).

What is GitOps?

GitOps is a way of operating infrastructure and applications where:

  1. Git is the single source of truth. Everything that should exist in your cluster is described declaratively in a git repository.
  2. Changes are made through pull requests. You don't kubectl apply ad hoc on a server; you propose a change, review it, and merge it.
  3. An operator reconciles the cluster to the repository. A tool watches the repository, computes the diff between "what is in git" and "what is running", and applies the missing pieces automatically.

If you have ever used tools like ArgoCD or Flux, you already know the pattern. But GitOps is not the same thing as "I keep some YAML files in a repo." The key is the continuous reconciliation loop: the running system is continuously pulled back toward the state described in git, not pushed there once.

The bootstrap problem

There is a chicken-and-egg problem at the heart of every GitOps setup:

Your cluster only trusts the git repository as its source of truth. But how does the cluster get configured to watch that repository in the first place?

That first push — the one that creates the repository, gives it credentials, and registers it with ArgoCD — is exactly the kind of step-before-the-loop work that GitOps tools deliberately do not do for you.

This is where a playbook like ansible/playbooks/seed-forgejo.yml comes in. Let's walk through what it does, because it is a compact, real-world example of everything that has to happen before GitOps can take over.

1. Create the git hosting

Forgejo is running inside the cluster behind a service (git/forgejo-http). The playbook:

  • waits for the deployment to be ready (kubectl rollout status),
  • opens a port-forward from 127.0.0.1:13000 to svc/forgejo-http:3000,
  • and then drives the Forgejo REST API over that tunnel.

Through the API it creates:

  • an organization kuber,
  • a restricted user dedicated to ArgoCD (least privilege: it exists only to read one repository),
  • a private repository kuber/kuber with main as default branch,
  • and grants that restricted user read collaborator access on the repo.

Every step is idempotent: it first does a GET, and only creates the object on a 404. You can run the playbook a hundred times; the tenth run is a no-op.

2. Register the repository with ArgoCD

ArgoCD stores repository credentials as Kubernetes Secrets. The playbook renders forgejo-repo-secret.yaml.j2, ensures the argocd namespace exists, and applies the Secret. Now ArgoCD knows it is allowed to pull from forgejo/kuber.

3. Push the seed commit

The clever bit. You do not want to seed the repository with a stale copy of the past. You want to seed it with the current state of your working tree — including uncommitted files, since the cluster already runs them.

The playbook builds a snapshot commit from the current working tree using plumbing commands:

git read-tree HEAD                    # start from where you are
git ls-files -z --cached --deleted \
  --modified --others --exclude-standard | xargs -0 git add --   # add everything,
                                                                 # even untracked
git write-tree                       # write the tree object
git commit-tree <tree> -p HEAD -m 'Seed Forgejo GitOps state'    # commit it

Then it force-pushes that commit to refs/heads/main on Forgejo.

Credentials never touch the shell history or the terminal: a temporary GIT_ASKPASS helper reads GIT_USERNAME/GIT_PASSWORD from environment variables (set in-process), and the whole task runs with no_log: true.

4. Verify and clean up

  • git ls-remote confirms refs/heads/main actually exists on the server.
  • The askpass helper and the temporary git index are deleted.
  • The port-forward is killed.

Now, and only now, ArgoCD has a repository to watch, and the reconciliation loop can take over.

Why this is a good example of the GitOps pros

Reproducibility

The entire bootstrap is a checked-in playbook. Any machine with kubectl, ansible, and vault secrets can recreate the flow. Nothing depends on a memory of "that time I clicked around in a web UI."

Idempotency and safety

Checks-before-creates and status_code: [200, 404] handling mean re-running is harmless. The least-privilege ArgoCD user, the restricted scope of the repo, and no_log on anything containing credentials are the security posture that GitOps encourages: automation accounts that can do exactly one job, nothing more.

Auditability

Every change the playbook makes — org, user, repo, collaborator, secret, seed commit — is a declared task in version control. You can review the pull request that added the playbook, and you can review the diff of the seed commit. That is the core GitOps promise: history is the change log.

The golden rule: no hand-run changes

Once this playbook has run, the way the cluster changes is: edit YAML → open PR → merge → ArgoCD syncs. The ad hoc kubectl apply becomes the exception, not the rule.

The cons — being honest about it

GitOps is not free. The same playbook is a great illustration of the costs.

The bootstrap is (necessarily) not GitOps

The playbook runs on localhost, uses kubectl, talks to the Forgejo API, and force-pushes a commit. All of that hand-held orchestration is the exact opposite of GitOps. You only get the reconciliation loop after you have painstakingly set up the pieces — and that setup itself was imperative, not declarative.

This is a real operational gap: your bootstrap cannot be expressed in a git repo that lives inside your cluster, because the repo must exist first. Tools like ArgoCD acknowledge this by making "register the first application" a manual-ish step. Plan for a small pile of imperative glue at the start.

Secrets management does not go away

Notice that the playbook needs vars/vault.yml and works with no_log. GitOps moves workloads into git, but secrets do not belong in git. You end up with a second system — SOPS, Vault, or an external secrets operator — and now you have to automate that too. The seed commit pushed by the playbook is exactly the kind of repository you must keep clean of credentials; leaks there are reproducible by anyone with repo access.

Reconciliation can fight the operator

Force-pushing +commit:refs/heads/main is powerful and dangerous. If a human kubectl applys something ad hoc, ArgoCD will dutifully undo it. If two engineers push conflicting states, the last sync wins. The loop is a strong convergence mechanism and it will not ask for permission. Drift is either impossible or a fight, depending on your point of view.

Operational complexity

You now operate:

  • the git host (Forgejo) inside the cluster (bootstrapping bootstrapping),
  • an extra control loop (ArgoCD) with its own web UI, credentials, and failure modes,
  • port-forward choreography, askpass helpers, temp indexes, vaults.

A small cluster that could be run by a shell script now has a small distributed system of automation on top of it. GitOps tools fail in interesting new ways: sync loops that flap, webhook/SSH authentication drift, and so on.

The "source of truth" is a myth in practice

A seed playbook proves it: the working tree it snapshots did not have a single authoritative representation. Real clusters accumulate state that git does not, or cannot, capture — persistent volume contents, DNS entries, LoadBalancer IP allocations, the router's DHCP reservations. GitOps keeps config in git; it does not make everyone tell the same truth about everything. You will still have drift in the things your operator does not watch.

The verdict

GitOps is the right default for most Kubernetes workloads, from your home cluster to a data center:

  • Use it when you have more than one machine, multiple environments you want to keep in lockstep, or a team that wants review and audit trails before change.
  • Think carefully when your state is mostly ephemeral experimentation, your secrets story is immature, or you cannot tolerate an operator that overrides hand-applied changes.

And when you adopt it, budget for the bootstrap. Write the glue that creates the org, the user, the repo, and the credentials — make it idempotent, make it least-privilege, make it no_log — and only then let the reconciliation loop take over. That is exactly what seed-forgejo.yml does, and it is a template worth copying for any project's GitOps on-ramp.

This article is based on the ansible/playbooks/seed-forgejo.yml playbook from the kuber repository.