Archive for the ‘Puppet’ Category

Loops and Variable Indirection in Puppet

Friday, September 3rd, 2010

A while back I wrote about writing some custom Puppet facts, one of which was to enumerate the enslaved interfaces for each bonded network interface on a Linux host. So for example your facter output might contain the following facts:

interfaces => bond0,eth0,eth1,eth2,sit0
ipaddress_bond0 => 192.168.0.1
ipaddress_eth2 => 10.0.0.1
slaves_bond0 => eth0,eth1

The bottom fact is my addition. Why do I even want that? Well, I have Nagios monitoring the host via the bonded interface so I’ll know if that breaks completely as the host will be down but I’d also like to know if any one of the enslaved interfaces are down as even though the host should still be up thanks to the bonded interface doing its magic, a cable might have been unplugged or something more serious might have happened. On a modern Linux host with the /sys filesystem, you can do the following to see if an interface is connected to hardware:

# cat /sys/class/net/eth0/carrier
1

That value will flip to 0 should the cable be pulled, etc. so I wrote a really basic Nagios test that exits with the OK or CRITICAL value based on that. Now to write the Puppet manifest, in english the logic sounds pretty simple:

“Loop through each interface reported by the $interfaces fact, if the interface looks like a bonded interface, (i.e. it matches /bond\d+/), then loop through each interface reported by the $slaves_$interface fact which should now exist and export a Nagios service definition for it.”

First problem, there’s no such thing as a loop in Puppet. You can get around this by using a definition:

define match_bonded_interfaces {
    if $name =~ /bond\d+/ {
        ...
    }
}
 
$array_of_interfaces = split($interfaces, ',')
 
match_bonded_interfaces { $array_of_interfaces: }

This will call the definition for each value in the array with $name being set to the current element each time.

The second problem that I hit after this was I needed to access my custom fact for the interface to enumerate the enslaved interfaces. Not wanting to hardcode the interface name and without thinking I had tried to do the following:

define match_bonded_interfaces {
    if $name =~ /bond\d+/ {
        $slaves = ${slaves_$name}
    }
}

It seemed semi-logical at the time until I actually came to try it out and found it didn’t at all do what I had hoped. After a few further failed attempts at guessing some undocumented syntax to do what I wanted, I gave up thinking there was no way to do variable indirection in Puppet, in other words, accessing a variable by the value of another variable.

To me this seemed kind of odd. After all, quite a lot of the facts reported by facter are structured like this:

list_of_things => key1,key2,key3
value_key1 => foo
value_key2 => bar
value_key3 => baz

And if there’s any trace of programmer in you, you’re going to want to access those in a way that scales anywhere between 1 and n keys, rather than hardcode each possible key value and the associated variable names. Something deep inside would stop me from committing something like the following:

if $interfaces =~ /\beth0\b/ {
    notice("IP address of eth0 is ${ipaddress_eth0}")
}
 
if $interfaces =~ /\beth1\b/ {
    notice("IP address of eth1 is ${ipaddress_eth1}")
}
 
etc.

But annoyingly, that was the only way I could utilise my new fact, and besides most of the servers only had one bonded interface, but it was still annoying nonetheless so I was pleasantly surprised when a friend pointed me towards this blog entry which offered a possible solution, you use Puppets inline_template function to evaluate a small ERB template fragment like so:

define monitor_interface {
    @@nagios_service {
        ...
    }
}
 
define match_bonded_interfaces {
    if $name =~ /bond\d+/ {
        $slave_fact = "slaves_${name}"
        $slaves = inline_template('<%= scope.lookupvar(slave_fact) %>')
        $array_of_slaves = split($slaves, ',')
        monitor_interface { $array_of_slaves: }
    }
}
 
$array_of_interfaces = split($interfaces, ',')
 
match_bonded_interfaces { $array_of_interfaces: }

I feel dirty now.

Custom Puppet Facts

Wednesday, July 28th, 2010

I really like the idea of using Puppet and similar tools for automating as much of server configuration as possible. As I slowly “puppet-ify” things I find the need to sometimes add custom facts to facter, the part of Puppet that provides information about the host system.

This would be easy were it not for the small problem that custom facts are written in Ruby, a language I’m not fluent in, although this gives me a reason to learn the basics. I’m familiar with various other languages so it shouldn’t be that hard, here’s what I’ve managed to cobble together so far…

The first fact returns a list of the Linux software RAID block devices on the host. This is just parsing the /proc/mdstat file that should exist on any Linux distribution:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
Facter.add('software_raid') do
  confine :kernel => :linux
  setcode do
    devices = []
    if FileTest.exists?('/proc/mdstat')
      File.open('/proc/mdstat', 'r') do |f|
        while line = f.gets
          if line =~ /^(md\d+)/
            devices.push($1)
          end
        end
      end
    end
    devices.sort.join(',')
  end
end

This next recipe creates one fact per bonded network interface on Linux, containing the list of enslaved interfaces. This is specific to CentOS, Fedora, Red Hat or any other similar distributions that use the /etc/sysconfig configuration files:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
require 'find'
 
Facter.value(:interfaces).split(',').each do |interface|
  if interface =~ /^bond\d+$/
    Facter.add("#{interface}_slaves") do
      confine :kernel => :linux, :operatingsystem => %w{CentOS Fedora RedHat}
      setcode do
        slaves = []
        Find.find('/etc/sysconfig/network-scripts') do |path|
          if FileTest.directory?(path)
            next
          else
            if path =~ /ifcfg-(.+)$/
              device = $1
              File.open(path, 'r') do |f|
                while line = f.gets
                  if line =~ /^MASTER\s*=\s*#{interface}$/
                    slaves.push(device)
                  end
                end
              end
            end
          end
        end
        slaves.sort.join(',')
      end
    end
  end
end

I use these two facts within my Puppet manifests as the basis for configuring additional Nagios tests to make sure these two pieces of functionality are working correctly.