GitOps Implementation: Lessons from Production

What GitOps Actually Means

GitOps uses Git as the source of truth for declarative infrastructure and applications. The core principles:

**Declarative**: Describe the desired state, not the steps to get there

**Versioned**: All changes tracked in Git

**Automated**: Agents ensure actual state matches desired state

**Reconciled**: Continuous correction of drift

Choosing Your Tool

The two main options:

ArgoCD: Better UI, application-centric model, good for teams that want visibility into deployments.

Flux: More composable, better for GitOps-native workflows, integrates well with other controllers.

Both work. Pick based on your team's preferences and existing tooling.

Repository Structure

How you organize Git repositories affects maintainability.

Monorepo: All configuration in one repository

Easier to manage initially

Can become unwieldy at scale

Simpler cross-cutting changes

Polyrepo: Configuration split across repositories

Better access control

More complexity in coordination

Scales better for large organizations

Hybrid: Application repos separate from infrastructure

Common pattern

Balances concerns

Start simple. Restructure when you hit actual problems.

Handling Secrets

Secrets in Git is the most common GitOps challenge. Options:

External Secrets Operator: Sync secrets from external stores (Vault, cloud providers)

SOPS: Encrypt files in Git with various key management backends

Pick one approach and standardize on it.

Managing Multiple Environments

Common patterns:

Directory per environment: Simple, explicit, some duplication

Kustomize overlays: Base configuration with environment-specific patches

Helm values files: If you're already using Helm

Avoid: Complex templating that makes it hard to understand what's deployed where.

Drift Detection and Reconciliation

GitOps agents detect when actual state differs from desired state. Configure:

Sync policies (automatic vs. manual)

Prune policies (delete resources removed from Git)

Health checks (when is a deployment successful?)

Start with manual sync to understand the workflow, then automate as confidence builds.

Common Challenges

Slow feedback: Changes in Git don't appear immediately. Set appropriate expectations.

Debugging failures: When sync fails, developers need visibility into why.

Emergency changes: Sometimes you need to change production immediately. Have a process that gets changes back into Git.

What GitOps Doesn't Solve

GitOps is an operational model, not a complete solution. You still need:

CI pipelines to build and test

Observability to understand what's running

Incident response processes