Elasticsearch clusters plugged into Cisco switches

May 31st, 2013

I was building up an Elasticsearch cluster as a storage backend for Logstash. Using the default Zen discovery method, I found that some of the nodes in the cluster could not automatically find each other.

Zen uses multicast to locate other nodes and by using tcpdump I could see that some nodes weren’t receiving the multicast traffic. This lead me straight to the network layer and I found the culprit; a Cisco Nexus 5000 switch.

Normally a switch often treats multicast traffic much like broadcast traffic, it floods the packet to every port on the same VLAN. However Cisco switches try to be clever and learn which ports are interested in receiving the traffic by listening for IGMP packets, (referred to as snooping), but IGMP is only sent if there’s a multicast router on the network, (note also I’m not trying to route multicast, all nodes are on the same VLAN).

The solution was to enable a feature on the Nexus known as an “IGMP querier”. What this does is mimic enough of a multicast router that nodes report which multicast groups they’re interested in receiving traffic for and the switch can then learn which ports to forward multicast traffic on.

On the Nexus 5000, I needed to add the following configuration:

vlan configuration 1234
  ip igmp snooping querier 1.2.3.4

(If you have a Catalyst switch running IOS, the configuration should be very similar)

The VLAN should match whatever VLAN the nodes are attached to, and you can basically make up the IP address used here, the switch sends IGMP packets with it as the source, but it’s never used as the destination for packets, nodes use a specific multicast group instead.

As soon as I added this configuration my Elasticsearch cluster sprung into life.

Soekris CPU Scaling and OpenBSD 5.2

December 25th, 2012

OpenBSD 5.2 was released nearly two months ago and I had forgotten to upgrade my Soekris net6501, apart from my driver now being part of the kernel, this release also added support to the SpeedStep frequency scaling driver for the Atom CPU.

The simple benchmark to keep in mind is using the md5(1) command, using its built-in time trial test. Under the previous OpenBSD 5.1 release:

# md5 -ttt
MD5 time trial.  Processing 1000000 10000-byte blocks...
Digest = f0843f04c524250749d014a8152920ec
Time   = 124.227876 seconds
Speed  = 80497230.750367 bytes/second

You can see I’m getting roughly 80 MB/s. Now upgrade to OpenBSD 5.2 and repeat the test, you’ll get pretty much the same speeds, however now we have the CPU frequency scaling to play with. The scaling is very coarse, the only two speeds supported are 600 MHz and whatever maximum the CPU supports, so in my case with a net6501-70, it’s 1.6 GHz. To change the scaling simply manipulate the hw.setperf sysctl(8) variable.

By default the CPU is running at 100%, confirmed by:

# sysctl hw.setperf
hw.setperf=100

Now set the CPU frequency to the slowest speed:

# sysctl hw.setperf=0
hw.setperf: 100 -> 0

Now running the test again I get the following results:

# md5 -ttt
MD5 time trial.  Processing 1000000 10000-byte blocks...
Digest = f0843f04c524250749d014a8152920ec
Time   = 123.948587 seconds
Speed  = 80678612.334645 bytes/second

Almost exactly the same speed, which doesn’t make sense given I’ve knocked 1 GHz off the clock speed! So put the CPU back to 100% and try again:

# sysctl hw.setperf=100
hw.setperf: 0 -> 100
# md5 -ttt
MD5 time trial.  Processing 1000000 10000-byte blocks...
Digest = f0843f04c524250749d014a8152920ec
Time   = 46.424699 seconds
Speed  = 215402581.285449 bytes/second

Ah-ha, about 210 MB/s this time! It transpires there’s a bug in the Soekris BIOS, despite advertising the CPU as 1.6 GHz it wasn’t programmed correctly and was only being clocked at 600 MHz, so all this time I’ve effectively had the base net6501-30 model albeit with the extra RAM. You can work around this by setting hw.setperf to 100 on each boot.

A new BIOS 1.41c has been released which fixes this issue and programs the CPU to run at its advertised maximum speed. However to upgrade to this involves my eternal battle with serial terminal software and uploading over XMODEM which is notoriously fickle, although I think I have it cracked…

I usually use a Mac OS X host with a KeySpan USB/Serial adapter to connect to the net6501 so I already have tools like cu(1) and screen(1). You’ll also need the lrzsz tools installed which if using MacPorts is as easy as:

# sudo port install lrzsz

Using cu(1), connect to the Soekris:

# sudo cu -l /dev/tty.KeySerial1 -s 19200

Power the board on, use Ctrl+P to break into the BIOS monitor and type download to start the Soekris waiting to receive over XMODEM. Now you need to type ~+sz -X /path/to/b6501_141c.bin, possibly as quickly as you can after the previous command. If that works, type flashupdate afterwards to reprogram the BIOS. You’ll get something like the following transcript:

> download
 
Start sending file using XMODEM/CRC protocol.
~+sz -X /path/to/b6501_141c.bin
Sending /path/to/b6501_141c.bin, 1982 blocks: Give your local XMODEM receive command now.
Bytes Sent: 253696   BPS:1746                            
 
Transfer complete
 
File downloaded succesfully, size 253696 Bytes.
 
> flashupdate
Updating BIOS Flash ,,,,,,,,,,,,, Done.
 
> reboot

Reboot and boot back into OpenBSD. Now the time trial should return a result of roughly 210 MB/s every time. Because I obviously don’t need 1.6 GHz of CPU all time, I’ve enabled the apmd(8) daemon which manipulates the hw.setperf variable based on the CPU idle time. Add the following to /etc/rc.conf.local:

apmd_flags="-C -f /dev/null"

The -f is only necessary when running i386 otherwise apmd(8) complains. Start with:

# /etc/rc.d/apmd start
apmd(ok)

Normally hw.setperf will be 0 however when you do something CPU-intensive (such as the MD5 time trial) apmd(8) will automatically adjust hw.setperf back to 100 so you still get the 210 MB/s result, but most of the time you’ll have lower power draw and less heat.

Soekris net6501 improvements for OpenBSD

June 9th, 2012

I’ve recently had committed to the OpenBSD project a kernel driver, tcpcib(4), that adds support for the hardware watchdog and HPET found in the Soekris net6501, (although it should also work on any system based around the Intel Atom E600 “Tunnel Creek” processor).

I’ve not had any issue with my net6501 that has required a hard reset although I managed to trigger the watchdog a few times on the net4501 by pushing too much network traffic through it, so it’s handy to have it there as a safeguard. The HPET support simply provides a time counter with better precision that benefits anything on the system that can use it.

With the driver in your kernel, you should now see the following attach lines:

tcpcib0 at pci0 dev 31 function 0 "Intel E600 LPC" rev 0x00: 14318179 Hz timer, watchdog
isa0 at tcpcib0

Enabling the watchdog is as easy as setting kern.watchdog.period with sysctl(8). The watchdog supports a maximum period of 600 seconds and if it ever triggers a reboot of the host, you’ll be treated to a slightly different attach line:

tcpcib0 at pci0 dev 31 function 0 "Intel E600 LPC" rev 0x00: 14318179 Hz timer, watchdog, reboot on timeout

The HPET support “just works”, but you can confirm it’s active with the following:

# sysctl kern.timecounter 
kern.timecounter.tick=1
kern.timecounter.timestepwarnings=0
kern.timecounter.hardware=tcpcib0
kern.timecounter.choice=i8254(0) tcpcib0(2000) dummy(-1000000)

The driver is available in -current and so should be in 5.2 onwards and can be backported to 5.1 which is how I’m currently running it.

HP Microserver Remote Management Card

January 28th, 2012

I recently acquired the Remote Management card for my HP Microserver, which allows remote KVM & power control, IPMI management and hardware monitoring through temperature & fan sensors.

Note the extra connector on the card in addition to the standard PCI-e x1 connector which matches the dedicated slot on the Microserver motherboard. This presented a bit of a problem as I was using the space for the battery backup module for the RAID controller in the neighbouring slot.

Thankfully the long ribbon cable meant I could route the battery up to the space behind the DVD burner freeing the slot again. Once the card was installed and everything screwed back together I booted straight back into CentOS. Given IPMI is touted as a feature I figured that was the first thing to try so I installed OpenIPMI:

# yum -y install OpenIPMI ipmitool
...
# service ipmi start
Starting ipmi drivers:                                     [  OK  ]
# ipmitool chassis status
Could not open device at /dev/ipmi0 or /dev/ipmi/0 or /dev/ipmidev/0: No such file or directory
Error sending Chassis Status command

Hmm, not good. Looking at dmesg shows the following is output when the IPMI drivers get loaded:

ipmi message handler version 39.2
IPMI System Interface driver.
ipmi_si: Adding SMBIOS-specified kcs state machine
ipmi_si: Adding ACPI-specified smic state machine
ipmi_si: Trying SMBIOS-specified kcs state machine at i/o address 0xca8, slave address 0x20, irq 0
ipmi_si: Interface detection failed
ipmi_si: Trying ACPI-specified smic state machine at mem address 0x0, slave address 0x0, irq 0
Could not set up I/O space
ipmi device interface

From reading the PDF manual it states that the IPMI KCS interface is at 0xCA2 in memory, not 0xCA8 that the kernel is trying to probe. Looking at the output from dmidecode shows where this value is probably coming from:

# dmidecode --type 38
# dmidecode 2.11
SMBIOS 2.6 present.
 
Handle 0x001B, DMI type 38, 18 bytes
IPMI Device Information
	Interface Type: KCS (Keyboard Control Style)
	Specification Version: 1.5
	I2C Slave Address: 0x10
	NV Storage Device: Not Present
	Base Address: 0x0000000000000CA8 (I/O)
	Register Spacing: Successive Byte Boundaries

This suggests a minor bug in the BIOS.

Querying the ipmi_si module with modinfo shows it can be persuaded to use a different I/O address so I created /etc/modprobe.d/ipmi.conf containing the following:

1
options ipmi_si type=kcs ports=0xca2

Then bounce the service to reload the modules and try again:

# service ipmi restart
Stopping all ipmi drivers:                                 [  OK  ]
Starting ipmi drivers:                                     [  OK  ]

As of CentOS 6.4, the above won’t work as the ipmi_si module is now compiled into the kernel. Instead, you need to edit /etc/grub.conf and append the following to your kernel parameters:

1
ipmi_si.type=kcs ipmi_si.ports=0xca2

Thanks to this post for the info. Instead of bouncing the service you’ll need to reboot, then try again:

# ipmitool chassis status
System Power         : on
Power Overload       : false
Power Interlock      : inactive
Main Power Fault     : false
Power Control Fault  : false
Power Restore Policy : always-off
Last Power Event     : 
Chassis Intrusion    : inactive
Front-Panel Lockout  : inactive
Drive Fault          : false
Cooling/Fan Fault    : false
# ipmitool sdr
Watchdog         | 0x00              | ok
CPU_THEMAL       | 32 degrees C      | ok
NB_THERMAL       | 35 degrees C      | ok
SEL Rate         | 0 messages        | ok
AMBIENT_THERMAL  | 20 degrees C      | ok
EvtLogDisabled   | 0x00              | ok
System Event     | 0x00              | ok
SYS_FAN          | 1000 RPM          | ok
CPU Thermtrip    | 0x00              | ok
Sys Pwr Monitor  | 0x00              | ok

Success! With that sorted, you can now use ipmitool to further configure the management card, although not all of the settings are accessible such as IPv6 network settings so you have to use the BIOS or web interface for some of it.

Overall, I’m fairly happy with the management card. It has decent IPv6 support and the Java KVM client works okay on OS X should I ever need it but I couldn’t coax the separate virtual media client to work, I guess only Windows is supported.

OpenBSD PPPoE and RFC 4638

January 20th, 2012

I upgraded my Internet connection from ADSL 2+ to FTTC a while ago. I’m with Eclipse as an ISP, but it’s basically the same product as BT Infinity, right down to the Openreach-branded modem, (a Huawei Echolife HG612 to be exact).

With this modem, you need to use a router or some software that can do RFC 2516 PPPoE so I simply kept using OpenBSD on my trusty Soekris net4501 and set up a pppoe(4) interface, job done. However what became apparent is the 133 MHz AMD Elan CPU couldn’t fully utilise the 40 Mb/s bandwidth I now had, at best I could get 16-20 Mb/s with a favourable wind. An upgrade was needed.

Given I’d had around 8 years of flawless service from the net4501, another Soekris board was the way to go. Enter the net6501 with comparatively loads more CPU grunt, RAM and interestingly Gigabit NIC chips; not necessarily for the faster speed, but because they can naturally handle a larger MTU.

The reason for this was that I had read that the Huawei modem and BT FTTC network fully supported RFC 4638, which means you can have an MTU of 1,500 bytes on your PPPoE connection which matches what you’ll have on your internal network. Traditionally a PPPoE connection only allowed 1,492 bytes on account of the overhead of 8 bytes of PPPoE headers in every Ethernet frame payload. Because of this it was almost mandatory to perform MSS clamping on traffic to prevent problems. So having an MTU of 1,500 bytes should avoid the need for any clamping trickery, but means your Ethernet interface needs to cope with an MTU of 1,508 bytes, hence the Gigabit NIC (which can accommodate an MTU of 9,000 bytes with no problems).

One small problem remained, pppoe(4) on OpenBSD 5.0 didn’t support RFC 4638. While I sat down and started to add support I noticed someone had added this to the NetBSD driver already, (which is where the OpenBSD driver originated from), so based on their changes I created a similar patch and with some necessary improvements based on feedback from OpenBSD developers it has now been committed to CVS in time for the 5.1 release.

To make use of the larger MTU is fairly obvious, simply set the MTU explicitly on both the Ethernet and PPPoE interfaces to 8 bytes higher than their default. As an example, my /etc/hostname.em0 now contains:

1
mtu 1508 up

And similarly my /etc/hostname.pppoe0 contains:

1
2
3
4
5
inet 0.0.0.0 255.255.255.255 NONE mtu 1500 \
        pppoedev em0 authproto chap \
        authname 'user' authkey 'secret' up
dest 0.0.0.1
!/sbin/route add default -ifp \$if 0.0.0.1

I also added support to tcpdump(8) to display the additional PPPoE tag used to negotiate the larger MTU, so when you bring the interface up, watch for PPP-Max-Payload tags going back and forth during the discovery phase.

With that done the remaining step is to remove any scrub max-mss rules in pf.conf(5) as with any luck they should no longer be required.

PPPoE fixes for natpmpd

December 16th, 2011

I recently started using the pppoe(4) driver on OpenBSD, and with it found a few small bugs in how natpmpd handles these sorts of dynamic interfaces.

One simple bug being it refused to start up if the interface didn’t already exist and also it considered 0.0.0.0 as a valid IP address and would broadcast that to any clients on the network. This situation happened due to the way pppoe(4) interfaces are initially set up and would correct itself quickly once the PPPoE session was negotiated.

Both of these bugs should be fixed so natpmpd should now correctly deny any request until the interface gets created and negotiates a normal IP address, (normal Ethernet interface behaviour should be unchanged).

Forcing GUID Partition Table on a disk with CentOS 6

August 6th, 2011

CentOS 6 is now out so I can finally build up a new HP ProLiant Microserver that I purchased for a new home server. Amongst many new features, CentOS 6 ships a GRUB bootloader that can boot from a disk with a GUID Partition Table (GPT for short). Despite not having EFI, the HP BIOS can boot from GPT disks so there’s no limitation with disk sizes that I can use.

However, before I tore down my current home server I wanted to test all of this really did work so I popped a “small” 500 GB disk in the HP. The trouble is the CentOS installer won’t use GPT if the disk is smaller than 2 TB. Normally this wouldn’t be a problem with a single disk apart from I want to specifically test GPT functionality but this can cause complications if your disk is in fact a hardware RAID volume because most RAID controllers allow the following scenario:

  1. You have a RAID 5 volume comprising 4x 500 GB disks, so the OS sees ~ 1.5 TB raw space.
  2. Pull one 500 GB disk, replace with a 1 TB disk, wait for RAID volume to rebuild.
  3. Rinse and repeat until all disks are upgraded.
  4. Grow RAID volume to use the extra space, OS now sees ~ 3 TB raw space.
  5. Resize partitions on the volume, grow filesystems, etc.

You’ll find your volume will have a traditional MBR partition scheme as it fell below the 2 TB limit so you can’t resize the partitions to fully use the new space. While it might be possible to non-destructively convert MBR to GPT, I wouldn’t want to risk it with terabytes of data. It would be far better to just use GPT from the start to save painting myself into a corner later.

If you’re doing a normal install, start the installer until you get to what’s known as stage 2 such that on (ctrl+)alt+F2 you have a working shell. From here, you can use parted to create an empty GPT on the disk:

# /usr/sbin/parted -s /dev/sda mklabel gpt

Return to the installer and keep going until you reach the partitioning choices. Pick the “use empty space” option and the installer will create new partitions within the new GPT. If you choose the “delete everything” option, the installer will replace the GPT with MBR again.

If like me you’re using kickstart to automate the install process, you can do something similar in your kickstart file with the %pre section, something like the following:

%pre
/usr/sbin/parted -s /dev/sda mklabel gpt
%end

Then make sure your kickstart contains no clearpart instructions so it will default to just using the empty space. The only small nit I found after the install is that the minimal install option only includes fdisk and not parted as well so if you want to manage the disk partitions you’ll need to add that either at install time or afterwards as fdisk doesn’t support GPT.

Storage controller probe order & Kickstart

June 14th, 2011

I’ve been configuring some Dell servers today that have one of Dell’s MD3220 storage arrays attached. The servers were completely blank so they needed kickstarting with CentOS 5.x. Not a problem, just use the standard one that’s used for 99% of the servers.

The problem that arises is the standard kickstart assumes the first disk it sees should be the disks internal to the server and partitions accordingly but annoyingly the installer loads the mpt2sas driver first for the HBA’s used to hook up the storage array, and then loads the megaraid_sas driver for the built-in RAID controller used for the internal disks. This means the internal disks could be anywhere from /dev/sdc onwards depending on what state the array is in.

An undocumented boot option is driverload= which seems to force the supplied list of drivers to be loaded first before anything else that’s detected. I found reference to this option in this post along with use of the nostorage boot option. However when I used the following options:

nostorage driverload=sd_mod:megaraid_sas

The installer still loaded the mpt2sas driver but it did force the megaraid_sas driver to be loaded first. This could just be a bug that mpt2sas isn’t on the list of “storage” drivers but the net result is acceptable in that /dev/sda now pointed to the internal disks. After the server rebooted /etc/modprobe.conf also had the correct scsi_hostadapterX aliases to keep things in the right order.

Using SNMP to upgrade IOS on Cisco devices

April 29th, 2011

Following on from my previous post regarding using SNMP to initiate TFTP backups, it’s also possible to use SNMP to remotely upgrade IOS on a Cisco device.

I’ll reuse and extend the setup and configuration from the previous article so apply that first. The IOS device then needs the following additions:

router(config)#snmp-server view myview ciscoFlashOps included
router(config)#snmp-server view myview lts.9 included
router(config)#snmp-server system-shutdown

The first command grants SNMP write access to allow us to manipulate the flash storage in the device. The remaining commands allow us to trigger a reload/reboot of the device remotely via SNMP.

On our “management station” we’ll need a few more MIB files installing under ~/mibs or wherever you put them:

  • CISCO-FLASH-MIB
  • ENTITY-MIB
  • OLD-CISCO-TS-MIB

Update the $MIBS environment variables to include them, you’ll need something like the following:

# export MIBS=+CISCO-CONFIG-COPY-MIB:CISCO-FLASH-MIB:OLD-CISCO-TS-MIB

Now, my devices don’t have enough flash storage to hold two IOS images so I normally have to delete the current version (which is loaded into memory, so it doesn’t affect the operation of the device) to free up space and then upload the new version, reload, and pray it works.

To find the filename of the current IOS image we can use the SNMP equivalent of dir flash:/:

# snmptable -v 3 -l authPriv -a MD5 -A authsecret -x DES -X privsecret -u myuser -Cb -Ci -Cw 80 192.0.2.254 ciscoFlashFileTable
SNMP table: CISCO-FLASH-MIB::ciscoFlashFileTable
 
 index           Size     Checksum Status                                Name
 1.1.1      576 bytes        "0x0"  valid                                   /
 1.1.2 16459360 bytes "0xDEADBEEF"  valid c870-advsecurityk9-mz.124-15.T9.bin
 
SNMP table CISCO-FLASH-MIB::ciscoFlashFileTable, part 2
 
 index      Type Date
 1.1.1 directory    ?
 1.1.2   unknown    ?

The index is a composite of the flash device, partition and file, so 1.1.2 is the second file on the first partition of the first flash device. Unless you have multiple flash devices and/or partitions you can largely ignore these details, take a look at ciscoFlashDeviceTable & ciscoFlashPartitionTable if you want more information.

With the filename, we now need to construct a request to delete it. This sort of operation along with tasks such as erasing the flash, etc. uses the ciscoFlashMiscOpTable SNMP table. Using the same technique I demonstrated in the previous article insert a new row and activate it:

# snmpset -v 3 -l authPriv -a MD5 -A authsecret -x DES -X privsecret -u myuser 192.0.2.254 \
ciscoFlashMiscOpCommand.23 i delete \
ciscoFlashMiscOpDestinationName.23 s c870-advsecurityk9-mz.124-15.T9.bin \
ciscoFlashMiscOpEntryStatus.23 i createAndGo
CISCO-FLASH-MIB::ciscoFlashMiscOpCommand.23 = INTEGER: delete(3)
CISCO-FLASH-MIB::ciscoFlashMiscOpDestinationName.23 = STRING: c870-advsecurityk9-mz.124-15.T9.bin
CISCO-FLASH-MIB::ciscoFlashMiscOpEntryStatus.23 = INTEGER: createAndGo(4)

Note that I’m using createAndGo instead of active for the row status because it’s part of the same initial set request. Now check the table to see if the operation was successful:

# snmptable -v 3 -l authPriv -a MD5 -A authsecret -x DES -X privsecret -u myuser -Cb -Ci -Cw 80 192.0.2.254 ciscoFlashMiscOpTableSNMP table: CISCO-FLASH-MIB::ciscoFlashMiscOpTable
 
 index Command                     DestinationName                 Status
    23  delete c870-advsecurityk9-mz.124-15.T9.bin miscOpOperationSuccess
 
SNMP table CISCO-FLASH-MIB::ciscoFlashMiscOpTable, part 2
 
 index NotifyOnCompletion         Time EntryStatus
    23              false 0:0:00:56.24      active

It worked so checking the flash storage should confirm the file is now absent. Now we can upload the new IOS image, you’ll need to have it copied to /tftpboot on your TFTP server. This uses another SNMP table but the procedure is exactly the same, insert a row describing the operation we want to perform:

# snmpset -v 3 -l authPriv -a MD5 -A authsecret -x DES -X privsecret -u myuser 192.0.2.254 \
ciscoFlashCopyCommand.23 i copyToFlashWithoutErase \
ciscoFlashCopyProtocol.23 i tftp \
ciscoFlashCopyServerAddress.23 a 192.0.2.1 \
ciscoFlashCopySourceName.23 s c870-advsecurityk9-mz.124-24.T4.bin \
ciscoFlashCopyEntryStatus.23 i createAndGo
CISCO-FLASH-MIB::ciscoFlashCopyCommand.23 = INTEGER: copyToFlashWithoutErase(2)
CISCO-FLASH-MIB::ciscoFlashCopyProtocol.23 = INTEGER: tftp(1)
CISCO-FLASH-MIB::ciscoFlashCopyServerAddress.23 = IpAddress: 192.0.2.1
CISCO-FLASH-MIB::ciscoFlashCopySourceName.23 = STRING: c870-advsecurityk9-mz.124-24.T4.bin
CISCO-FLASH-MIB::ciscoFlashCopyEntryStatus.23 = INTEGER: createAndGo(4)

Chances are this operation will take a minute or two to complete, keep checking ciscoFlashCopyTable until it completes:

# snmptable -v 3 -l authPriv -a MD5 -A authsecret -x DES -X privsecret -u myuser -Cb -Ci -Cw 80 192.0.2.254 ciscoFlashCopyTable
SNMP table: CISCO-FLASH-MIB::ciscoFlashCopyTable
 
 index                 Command Protocol ServerAddress
    23 copyToFlashWithoutErase     tftp     192.0.2.1
 
SNMP table CISCO-FLASH-MIB::ciscoFlashCopyTable, part 2
 
 index                          SourceName DestinationName RemoteUserName
    23 c870-advsecurityk9-mz.124-24.T4.bin                               
 
SNMP table CISCO-FLASH-MIB::ciscoFlashCopyTable, part 3
 
 index               Status NotifyOnCompletion         Time EntryStatus Verify
    23 copyOperationSuccess              false 0:0:05:03.71      active  false
 
SNMP table CISCO-FLASH-MIB::ciscoFlashCopyTable, part 4
 
 index ServerAddrType ServerAddrRev1 RemotePassword
    23           ipv4    "192.0.2.1"              ?

With the new image uploaded it’s time for the coup de grĂ¢ce, reloading the device:

# snmpset -v 3 -l authPriv -a MD5 -A authsecret -x DES -X privsecret -u myuser 192.0.2.254 \
tsMsgSend.0 i reload
OLD-CISCO-TS-MIB::tsMsgSend.0 = INTEGER: reload(2)

Your device should now reboot and after a short while come back up running the updated IOS image. You can confirm the source of the reboot request with the following:

router#show version | i reason
Last reload reason: snmp shutdown request

Powerful stuff. Needless to say, don’t make this accessible to the world with an SNMPv2 community of “private”.

Using SNMP to trigger Cisco TFTP backups

April 24th, 2011

I recently discovered you can use SNMP to trigger a Cisco device to write its configuration to a remote TFTP server. Apparently this feature has existed in Cisco devices in one form or another for over a decade, but somehow I’d missed it completely. This seems a far better alternative to getting in a muddle with Expect-style scripts to telnet or SSH into the device. One or more Access Control Lists can also be utilised to restrict access.

First of all, let’s configure the IOS device which will need to be running at least IOS 12.0. The first thing to do is create an Access Control List which matches the IP address of the “management station” which will be issuing the SNMP commands and for simplicities sake is also the TFTP server:

router(config)#access-list 20 permit host 192.0.2.1
router(config)#access-list 20 deny any

Next we define an SNMP view that restricts how much of the tree is writable, allowing write access to everything is both unnecessary and inadvisable:

router(config)#snmp-server view myview ccCopyTable included

Now we create an SNMP group that ties the ACL and view together:

router(config)#snmp-server group mygroup v3 priv write myview access 20

Note that I’ve specified SNMPv3 along with the authPriv security level, not all devices support that so adjust accordingly. I’ve also not specified a read view so it will use the default v1default view which can read pretty much everything. Next create a new user as a member of this group:

router(config)#snmp-server user myuser mygroup v3 auth md5 authsecret priv des privsecret

Finally, restrict which TFTP servers can be written to, triggered by SNMP. This stops someone from making your device write files to an arbitrary TFTP server somewhere:

router(config)#snmp-server tftp-server-list 20

That should be all of the configuration needed for the Cisco device.

Now on our “management station” which can be anything provided it has a recent version of Net-SNMP installed, (CentOS 5.x works fine), first make sure TFTP is installed and running. Normally the TFTP server cannot create new files so create an empty file ready:

# touch /tftpboot/router.conf

Now install the various MIB files we’ll need. You don’t need these but for the sake of readability it’s easier to use the symbolic names rather than raw OID strings. You’ll need to download the following MIB files:

  • CISCO-CONFIG-COPY-MIB
  • CISCO-SMI
  • CISCO-ST-TC

Put them in a directory such as ~/mibs and then configure Net-SNMP with the following two environment variables:

# export MIBDIRS=+${HOME}/mibs
# export MIBS=+CISCO-CONFIG-COPY-MIB

You should be able to test that everything is configured correctly:

# snmptable -v 3 -l authPriv -a MD5 -A authsecret -x DES -X privsecret -u myuser 192.0.2.254 ccCopyTable
CISCO-CONFIG-COPY-MIB::ccCopyTable: No entries

Now we need to create a row in this table describing the copy operation we’d like to perform. Without getting too heavy into how SNMP tables work, we basically need to use a unique index number to refer to the row. As we’ve already shown, the table is empty so pick any random number, umm, 23! Create a new row with the following column values:

# snmpset -v 3 -l authPriv -a MD5 -A authsecret -x DES -X privsecret -u myuser 192.0.2.254 \
> ccCopyProtocol.23 i tftp \
> ccCopySourceFileType.23 i runningConfig \
> ccCopyDestFileType.23 i networkFile \
> ccCopyServerAddress.23 a 192.0.2.1 \
> ccCopyFileName.23 s router.conf
CISCO-CONFIG-COPY-MIB::ccCopyProtocol.23 = INTEGER: tftp(1)
CISCO-CONFIG-COPY-MIB::ccCopySourceFileType.23 = INTEGER: runningConfig(4)
CISCO-CONFIG-COPY-MIB::ccCopyDestFileType.23 = INTEGER: networkFile(1)
CISCO-CONFIG-COPY-MIB::ccCopyServerAddress.23 = IpAddress: 192.0.2.1
CISCO-CONFIG-COPY-MIB::ccCopyFileName.23 = STRING: router.conf

Now if you re-run the snmptable command (with some extra flagsoup to help readability) you’ll see there’s now a row:

# snmptable -v 3 -l authPriv -a MD5 -A authsecret -x DES -X privsecret -u myuser -Cb -Ci -Cw 80 192.0.2.254 ccCopyTable
SNMP table: CISCO-CONFIG-COPY-MIB::ccCopyTable
 
 index Protocol SourceFileType DestFileType ServerAddress    FileName UserName
    23     tftp  runningConfig  networkFile     192.0.2.1 router.conf        ?
 
SNMP table CISCO-CONFIG-COPY-MIB::ccCopyTable, part 2
 
 index UserPassword NotificationOnCompletion State TimeStarted TimeCompleted
    23            ?                    false     ?           ?             ?
 
SNMP table CISCO-CONFIG-COPY-MIB::ccCopyTable, part 3
 
 index FailCause EntryRowStatus ServerAddressType ServerAddressRev1
    23         ?              ?              ipv4       "192.0.2.1"

This won’t initiate the operation until we set one more column to activate the table row:

# snmpset -v 3 -l authPriv -a MD5 -A authsecret -x DES -X privsecret -u myuser 192.0.2.254 \
> ccCopyEntryRowStatus.23 i active
CISCO-CONFIG-COPY-MIB::ccCopyEntryRowStatus.23 = INTEGER: active(1)

Now check the table again:

# snmptable -v 3 -l authPriv -a MD5 -A authsecret -x DES -X authsecret -u myuser -Cb -Ci -Cw 80 192.0.2.254 ccCopyTable
SNMP table: CISCO-CONFIG-COPY-MIB::ccCopyTable
 
 index Protocol SourceFileType DestFileType ServerAddress    FileName UserName
    23     tftp  runningConfig  networkFile     192.0.2.1 router.conf        ?
 
SNMP table CISCO-CONFIG-COPY-MIB::ccCopyTable, part 2
 
 index UserPassword NotificationOnCompletion      State  TimeStarted
    23            ?                    false successful 5:0:20:49.92
 
SNMP table CISCO-CONFIG-COPY-MIB::ccCopyTable, part 3
 
 index TimeCompleted FailCause EntryRowStatus ServerAddressType
    23  5:0:20:51.39         ?         active              ipv4
 
SNMP table CISCO-CONFIG-COPY-MIB::ccCopyTable, part 4
 
 index ServerAddressRev1
    23       "192.0.2.1"

You can see the state column now reads successful so check the empty file you created under /tftpboot, it should now have the contents of the current configuration for your device.

The row in the table will be automatically removed after a few minutes but you can also explicitly remove it:

# snmpset -v 3 -l authPriv -a MD5 -A authsecret -x DES -X privsecret -u myuser 192.0.2.254 \
> ccCopyEntryRowStatus.23 i destroy
CISCO-CONFIG-COPY-MIB::ccCopyEntryRowStatus.23 = INTEGER: destroy(6)
# snmptable -v 3 -l authPriv -a MD5 -A authsecret -x DES -X privsecret -u myuser 192.0.2.254 ccCopyTable
CISCO-CONFIG-COPY-MIB::ccCopyTable: No entries

There’s still a fair bit of functionality that I’ve skipped over, you might be able to use something other than TFTP or generate an SNMP trap on completion. Experiment with changing the Access Control Lists and stopping the TFTP server to see how the table output changes.