Bootstrapping Infrastructure: VCS for TF State

Introduction

I use OpenTofu/Terraform(TF)¹ to provision/configure my infrastructure, etc. - mostly because an “Infrastructure As Code”² approach is a sine qua non for me³.

• If I use an “Infrastructure As Code”/“Everything as Code” approach for all my projects in order to have programmatic control over all the things, that should include my VCS repositories, right?
• TF⁴ needs to store a State file⁵. The State file contains information about the current state of the Resources that TF wants to manage in order to determine what needs to be changed.
• In the default/trivial/minimal case, TF will use a local file to store State which works but has limitations; what happens when I want to work with this State from a different machine? Or if there is a team? Should I sneakernet it? No, of course not. And so all these technologies include functionality that supports remote State, and most of them support different kinds of backends for storing these States.
• But these backends are infrastructure. So I should manage them with my IaC approach. So now we see a bootstrapping problem.

But git is a distributed VCS, so how about I⁶ use git as my backend for TF State? That way I can IaC my IaC; this is my usage of terraform-backend-git from Dee Kryvenko. It is a TF Backend provider that allows TF to store its State files in a git repository. See the the author’s explanation for some reasoning.

What are the goals?

Starting with only an account at a VCS Forge (github/gitlab/forgejo/etc.) and a token to allow programmatic access, I want:

• A git repository that contains/can contain TF State files with a conventional directory structure that makes sense for my TF projects.
• A simple workflow/process so that my TF projects all use the same approach for finding/using their TF State.

Implementation

Each TF project(“configuration”) has a root module, with persisted state.

TF projects have their persistent state stored in a directory structure in the tf-states repository that uses the following pattern:

tf-states
└── Project Group
    └── the Project's (primary) TF provider

For example:

tf-states
├── rh
│   └── github
│   └── forgejo
└── swip
    ├── aws
    ├── github
    └── gitlab

Usage

Upon starting⁷ a new TF project, add a new HCL file, tf-backend-git.hcl⁸:

git.repository = "git@github.com:robinhutty/tf-states.git"
# git.repository = "https://github.com/robinhutty/tf-states"
git.ref = "main"
# The path to the `state` below should follow the directory structure above
git.state = "rh/github/state.json"

Then use terraform-backend-git as a wrapper for terragrunt⁹/opentofu/terraform:

$ terraform-backend-git git terraform -t $(command -v tofu) init

How to bootstrap?

1. Create a local git repo, <path>/rh/github, that will contain the TF project that manages GitHub for the rh effort - at least a root module, but it can call child module(s) if they’re available. Might as well use boilerplate, eh?

   # for a TF project named `foo`
   git init
   echo "# foo" > README.md
   git add README.md
   git commit -m "Init foo"
   git branch -M main
   git remote add origin git@github.com:robinhutty/foo.git
   git push -u origin main
   
2. Write TF code to provision/configure the set of GitHub repositories and other resources for the rh effort.
3. At a minimum, that set should contain a repository for the tf-states! But to make this non-trivial, I’ll also add another repository for my (child) tf-modules.
4. Run a TF init - which will install GitHub provider, etc.
5. Run a TF apply - which will provision/configure the GitHub repositories
6. Add a tf-backend-git.hcl referring to the new tf-states repository and the path to where the State file within the tf-states repository should live.
7. Run a TF init -migrate-state.
8. Run a TF plan - which should show no changes.

• create a local git repo, tf-states, with the directory structure I want to use to store the TF State for my TF projects

• add a readme, commit

• use terraform-backend-git to provision (at least) the tf-states repository:

  $ terraform-backend-git git \
      --repository /path/to/tf-states \
      --ref main \
      --state rhu/tf/github/state.json \
      terraform -t $(command -v tofu) init
  $ <snip> plan
  $ <snip> apply

• add the new (remote/GitHub) repository as an origin and push to it

     git remote add origin git@github.com:robinhutty/tf-states.git
     git push -u origin main

• migrate the state into the newly configured repository

  $ terraform-backend-git git \
      --repository git@github.com:robinhutty/tf-states.git \
      --ref main \
      --state rhu/tf/github/state.json \
      terraform -t $(command -v tofu) init -migrate-state