Checking for Unavailable Servers with Consul Template

Photo by Tanner Boriack on Unsplash

The Mission

Create a list of unavailable servers (nodes) from data provided by Hashicorp Consul.

The Problem

Consul’s API, Commands (CLI), DNS and Template features make it easy to get the locations and statuses of services running across a cluster of servers.

Here’s a cool-looking picture of some electronic squiggles to help us deal with that fact. Photo by hao wang on Unsplash

A Solution:

One approach is to use consul-template to create and regularly update a list of active nodes and then diff that with a list of expected nodes provided by an infrastructure automation tool like Chef.

consul-template \
-template "/var/test/actual_nodes.ctmpl:/var/test/actual_nodes.txt"
{{range datacenters}}{{range nodes (printf "@%s" .)}}{{.Node}}
{{end}}{{end}}
node.nodes.map{|n| n[:fqdn].split('.').first}.join("\n") + "\n"
node1
node2
node3
diff /tmp/expected_nodes <(sort /tmp/active_nodes)
diff \
— old-line-format="%L" \
— new-line-format="" \
— unchanged-line-format="" \
expected_nodes.txt <(sort active_nodes.txt)
node1
node3
node1
node2
node3
node2

A Check

To round up, here’s an idea for a bash script that could be used in a consul check (ideally run on multiple nodes). We just need to pass it the paths to the files containing lists of expected and currently-active nodes:

#!/bin/bashUSAGE="$(basename $0) expected_path active_path"if [ $# -lt 2 ]; then echo "$USAGE" && exit 1; fiFILE_DIFF="$(diff \
--old-line-format="%L" \
--new-line-format="" \
--unchanged-line-format="" \
"$1" <(sort "$2"))"

if [ -z "$FILE_DIFF" ]; then
echo "all nodes are live"
exit 0
fi
echo "unavailable nodes:"
echo "$FILE_DIFF"
exit 2

I like helping people to discover their own potential. He/him. Full-time parent & software developer, part-time teacher & musician.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store