Quick disclaimer: this article is about the Chef configuration management tool. It makes broad assumptions that the reader already has a good understanding of both this and git version control.

Why do we need serverless Chef?

Our team kept bumping up against one of these limits. While developers worked simultaneously on multiple git branches, our Chef-server stored a single set of configurations. These were used for all our servers.

Since we uploaded changes to the chef-server cookbook by cookbook and role by role, often from feature branches, it was easy for the data on our chef-server to get out of sync with what was in github. We thought of locking the chef-server to our ‘main’ branch by allowing uploads to the chef-server only through an automated CD (Continuous Deployment) process but this prevented us from being able to quickly test changes to cookbooks on individual nodes in the production environment. I talso prevented us from making pull-requests for use in our code-review and testing process.

What we all really wanted was an ability to apply git branches to individual nodes. This too has its dangers and pitfalls. It’s possible for another git branch to get behind the main branch and miss out on critical updates. We also would lose the certainty that all servers were provisioned using the same configuration. We decided that, with some additional monitoring to show where we had deployed a branch other than ‘main,’ these issues were worth the gains we would have from finding a way to run serverless chef.

What did we have to overcome?

  • node configurations (knife node)
  • cookbooks and roles (knife path-based commands)
  • finding servers by role and recipe (knife search)
  • performing remote operations on multiple servers (knife ssh)

In order to run without a chef-server, we had to find relatively stable workarounds for each of these points. Here’s how we handled them.

Node Configurations

What we did need was a way of storing an association between a node and its role. We already had a distributed key/value (k/v) store (HashiCorp Consul) installed on all nodes in our infrastructure, so we decided to store the association there. This allowed nodes to know about the role of every other node.

We then needed to load the data into a format that our existing chef recipes would work with. We created a custom recipe that would run on all nodes, loading data about other nodes from consul and creating the objects for Chef to use. Here’s a quick peek at part of that process:

Node.new.tap do |node|
node.run_list(['role[loaded_from_kv_store]'])
node.expand!
end

The k/v store also allowed us to store a JSON object for each node, with data such as custom ports. These would be useful when writing configurations on other nodes. Hopefully in future, we’ll be able to rewrite recipes in a way that requires less data to be stored here. This could be achieved by sourcing port info from registered Consul services, but that’s something for another post.

Cookbooks and Roles

  • Uses a deploy key to pull down a shallow clone of a particular branch of our chef git repo into a temp dir (single-branch mode with a depth of 1).
  • Reads data from locally written config files and writes a simple JSON config file for Chef to use.
  • Calls chef-client from the cloned directory with the “local,” “node-name” and “json-attributes” flags.

Finding servers by role and recipe

Performing remote options on multiple servers

Chickens and Eggs

Bootstrapping

When the Key/Value Store Is Unavailable

Creating New Roles

We eventually decided that the simplest way to handle this was that unknown roles would be considered equivalent to the base role. This allowed chef-client to keep running when a role was only available on a single branch.

Migrating Legacy Nodes

Bonus Features we Added in Later

  • Log level. Passing this on to chef-client made debugging much easier.
  • Process lock. Of course, chef-client has its own run lock, but adding one to our script ensured that the repo doesn’t get pulled again when the script is already running. We used flock for this.
  • Avoiding pulling the same code repeatedly. By comparing hashes of the local and github heads we were able to only pull when changes exist in the github repo.

In Conclusion

I hope this article helps you and I wish you all the best on your own adventures with serverless-chef.

I like helping people to discover their own potential. He/him. Full-time parent & software developer, part-time teacher & musician.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store