Bodgit & Scarper

HP Microserver TPM

January 10th, 2014

I’ve been dabbling with DNSSEC which involves creating a few zone- and key-signing keys, and it became immediately apparent that my headless HP Microserver has very poor entropy generation for /dev/random. After poking and prodding it became apparent there’s no dormant hardware RNG that I can just enable to fix it.

Eventually I stumbled on this post which suggests you can install and make use of the optional TPM as a source of entropy.

I picked up one cheaply and installed it following the above instructions to install and configure it; I found I only needed to remove the power cord for safety’s sake, the TPM connector on the motherboard is right at the front so I didn’t need to pull the tray out.

Also, since that blog post, the rng-tools package on RHEL/CentOS 6.x now includes an init script so it’s just a case of doing the following final step:

# chkconfig rngd on
# service rngd start

It should then be possible to pass this up to any KVM guests using the virtio-rng.ko module.

Tags: Entropy, HP, Microserver, TPM
Posted in CentOS | No Comments »

IPsec between Meraki and OpenBSD

August 6th, 2013

I recently acquired some (Cisco) Meraki networking kit including an MX60 security appliance (read: router, firewall, NAT, etc.).

Once it’s set up and running, I was browsing the dashboard and the site-to-site VPN configuration options. Normally with multiple Meraki devices in use, a fully-meshed VPN can be created automatically with very little configuration.

I also noticed the ability to add non-Meraki VPN peers so I added details for my OpenBSD-based gateway. You can see from the screenshot the details are basic:

On the OpenBSD side, I started with a basic configuration in /etc/ipsec.conf:

ike esp from 192.0.2.0/24 to 192.168.128.0/24 \
        peer example-xyzzy.dynamic-m.com \
        psk mekmitasdigoat

The Meraki dashboard allows you to register a dynamic DNS record for the MX60 which will be in the .dynamic-m.com domain so you can use this to refer to the remote peer in the configuration, (especially if you have a dynamic IP address).

This configuration didn’t work, but using isakmpd(8)‘s handy -L option to write to /var/run/isakmpd.pcap all of the unencrypted IKE traffic it then becomes apparent the OpenBSD and the Meraki disagree on the phase 1 negotiation; mainly that the Meraki end wants to use Triple DES and OpenBSD prefers AES by default. As there’s no way to tweak the cryptography options on the Meraki side, we change the OpenBED side like so:

ike esp from 192.0.2.0/24 to 192.168.128.0/24 \
        peer example-xyzzy.dynamic-m.com \
        main auth hmac-sha1 enc 3des group modp1024 lifetime 28800 \
        quick auth hmac-sha1 enc aes-256 group none lifetime 28800 \
        psk mekmitasdigoat

The main option sets the phase 1 parameters and the quick option sets the phase 2 parameters that match the highest settings out of the handful proposed by the Meraki side.

With that done, all that remains is to ensure isakmpd(8) starts at boot and the rules in /etc/ipsec.conf are automatically loaded by adding the following to /etc/rc.conf.local:

isakmpd_flags=-K
ipsec=YES

Tags: Cisco, ipsec, Meraki, MX60, VPN
Posted in Networks, OpenBSD | 1 Comment »

Elasticsearch clusters plugged into Cisco switches

May 31st, 2013

I was building up an Elasticsearch cluster as a storage backend for Logstash. Using the default Zen discovery method, I found that some of the nodes in the cluster could not automatically find each other.

Zen uses multicast to locate other nodes and by using tcpdump I could see that some nodes weren’t receiving the multicast traffic. This lead me straight to the network layer and I found the culprit; a Cisco Nexus 5000 switch.

Normally a switch often treats multicast traffic much like broadcast traffic, it floods the packet to every port on the same VLAN. However Cisco switches try to be clever and learn which ports are interested in receiving the traffic by listening for IGMP packets, (referred to as snooping), but IGMP is only sent if there’s a multicast router on the network, (note also I’m not trying to route multicast, all nodes are on the same VLAN).

The solution was to enable a feature on the Nexus known as an “IGMP querier”. What this does is mimic enough of a multicast router that nodes report which multicast groups they’re interested in receiving traffic for and the switch can then learn which ports to forward multicast traffic on.

On the Nexus 5000, I needed to add the following configuration:

vlan configuration 1234
  ip igmp snooping querier 1.2.3.4

(If you have a Catalyst switch running IOS, the configuration should be very similar)

The VLAN should match whatever VLAN the nodes are attached to, and you can basically make up the IP address used here, the switch sends IGMP packets with it as the source, but it’s never used as the destination for packets, nodes use a specific multicast group instead.

As soon as I added this configuration my Elasticsearch cluster sprung into life.

Tags: Catalyst, Cisco, Elasticsearch, IOS, Logstash, Multicast, Nexus, NX-OS
Posted in Networks | No Comments »

Soekris CPU Scaling and OpenBSD 5.2

December 25th, 2012

OpenBSD 5.2 was released nearly two months ago and I had forgotten to upgrade my Soekris net6501, apart from my driver now being part of the kernel, this release also added support to the SpeedStep frequency scaling driver for the Atom CPU.

The simple benchmark to keep in mind is using the md5(1) command, using its built-in time trial test. Under the previous OpenBSD 5.1 release:

# md5 -ttt
MD5 time trial.  Processing 1000000 10000-byte blocks...
Digest = f0843f04c524250749d014a8152920ec
Time   = 124.227876 seconds
Speed  = 80497230.750367 bytes/second

You can see I’m getting roughly 80 MB/s. Now upgrade to OpenBSD 5.2 and repeat the test, you’ll get pretty much the same speeds, however now we have the CPU frequency scaling to play with. The scaling is very coarse, the only two speeds supported are 600 MHz and whatever maximum the CPU supports, so in my case with a net6501-70, it’s 1.6 GHz. To change the scaling simply manipulate the hw.setperf sysctl(8) variable.

By default the CPU is running at 100%, confirmed by:

# sysctl hw.setperf
hw.setperf=100

Now set the CPU frequency to the slowest speed:

# sysctl hw.setperf=0
hw.setperf: 100 -> 0

Now running the test again I get the following results:

# md5 -ttt
MD5 time trial.  Processing 1000000 10000-byte blocks...
Digest = f0843f04c524250749d014a8152920ec
Time   = 123.948587 seconds
Speed  = 80678612.334645 bytes/second

Almost exactly the same speed, which doesn’t make sense given I’ve knocked 1 GHz off the clock speed! So put the CPU back to 100% and try again:

# sysctl hw.setperf=100
hw.setperf: 0 -> 100
# md5 -ttt
MD5 time trial.  Processing 1000000 10000-byte blocks...
Digest = f0843f04c524250749d014a8152920ec
Time   = 46.424699 seconds
Speed  = 215402581.285449 bytes/second

Ah-ha, about 210 MB/s this time! It transpires there’s a bug in the Soekris BIOS, despite advertising the CPU as 1.6 GHz it wasn’t programmed correctly and was only being clocked at 600 MHz, so all this time I’ve effectively had the base net6501-30 model albeit with the extra RAM. You can work around this by setting hw.setperf to 100 on each boot.

A new BIOS 1.41c has been released which fixes this issue and programs the CPU to run at its advertised maximum speed. However to upgrade to this involves my eternal battle with serial terminal software and uploading over XMODEM which is notoriously fickle, although I think I have it cracked…

I usually use a Mac OS X host with a KeySpan USB/Serial adapter to connect to the net6501 so I already have tools like cu(1) and screen(1). You’ll also need the lrzsz tools installed which if using MacPorts is as easy as:

# sudo port install lrzsz

Using cu(1), connect to the Soekris:

# sudo cu -l /dev/tty.KeySerial1 -s 19200

Power the board on, use Ctrl+P to break into the BIOS monitor and type download to start the Soekris waiting to receive over XMODEM. Now you need to type ~+sz -X /path/to/b6501_141c.bin, possibly as quickly as you can after the previous command. If that works, type flashupdate afterwards to reprogram the BIOS. You’ll get something like the following transcript:

> download
 
Start sending file using XMODEM/CRC protocol.
~+sz -X /path/to/b6501_141c.bin
Sending /path/to/b6501_141c.bin, 1982 blocks: Give your local XMODEM receive command now.
Bytes Sent: 253696   BPS:1746                            
 
Transfer complete
 
File downloaded succesfully, size 253696 Bytes.
 
> flashupdate
Updating BIOS Flash ,,,,,,,,,,,,, Done.
 
> reboot

Reboot and boot back into OpenBSD. Now the time trial should return a result of roughly 210 MB/s every time. Because I obviously don’t need 1.6 GHz of CPU all time, I’ve enabled the apmd(8) daemon which manipulates the hw.setperf variable based on the CPU idle time. Add the following to /etc/rc.conf.local:

apmd_flags="-C -f /dev/null"

The -f is only necessary when running i386 otherwise apmd(8) complains. Start with:

# /etc/rc.d/apmd start
apmd(ok)

Normally hw.setperf will be 0 however when you do something CPU-intensive (such as the MD5 time trial) apmd(8) will automatically adjust hw.setperf back to 100 so you still get the 210 MB/s result, but most of the time you’ll have lower power draw and less heat.

Tags: atom, BIOS, net6501, Soekris
Posted in OpenBSD | 10 Comments »

Soekris net6501 improvements for OpenBSD

June 9th, 2012

I’ve recently had committed to the OpenBSD project a kernel driver, tcpcib(4), that adds support for the hardware watchdog and HPET found in the Soekris net6501, (although it should also work on any system based around the Intel Atom E600 “Tunnel Creek” processor).

I’ve not had any issue with my net6501 that has required a hard reset although I managed to trigger the watchdog a few times on the net4501 by pushing too much network traffic through it, so it’s handy to have it there as a safeguard. The HPET support simply provides a time counter with better precision that benefits anything on the system that can use it.

With the driver in your kernel, you should now see the following attach lines:

tcpcib0 at pci0 dev 31 function 0 "Intel E600 LPC" rev 0x00: 14318179 Hz timer, watchdog
isa0 at tcpcib0

Enabling the watchdog is as easy as setting kern.watchdog.period with sysctl(8). The watchdog supports a maximum period of 600 seconds and if it ever triggers a reboot of the host, you’ll be treated to a slightly different attach line:

tcpcib0 at pci0 dev 31 function 0 "Intel E600 LPC" rev 0x00: 14318179 Hz timer, watchdog, reboot on timeout

The HPET support “just works”, but you can confirm it’s active with the following:

# sysctl kern.timecounter 
kern.timecounter.tick=1
kern.timecounter.timestepwarnings=0
kern.timecounter.hardware=tcpcib0
kern.timecounter.choice=i8254(0) tcpcib0(2000) dummy(-1000000)

The driver is available in -current and so should be in 5.2 onwards and can be backported to 5.1 which is how I’m currently running it.

Tags: atom, net6501, Soekris, watchdog
Posted in OpenBSD | No Comments »

HP Microserver Remote Management Card

January 28th, 2012

I recently acquired the Remote Management card for my HP Microserver, which allows remote KVM & power control, IPMI management and hardware monitoring through temperature & fan sensors.

Note the extra connector on the card in addition to the standard PCI-e x1 connector which matches the dedicated slot on the Microserver motherboard. This presented a bit of a problem as I was using the space for the battery backup module for the RAID controller in the neighbouring slot.

Thankfully the long ribbon cable meant I could route the battery up to the space behind the DVD burner freeing the slot again. Once the card was installed and everything screwed back together I booted straight back into CentOS. Given IPMI is touted as a feature I figured that was the first thing to try so I installed OpenIPMI:

# yum -y install OpenIPMI ipmitool
...
# service ipmi start
Starting ipmi drivers:                                     [  OK  ]
# ipmitool chassis status
Could not open device at /dev/ipmi0 or /dev/ipmi/0 or /dev/ipmidev/0: No such file or directory
Error sending Chassis Status command

Hmm, not good. Looking at dmesg shows the following is output when the IPMI drivers get loaded:

ipmi message handler version 39.2
IPMI System Interface driver.
ipmi_si: Adding SMBIOS-specified kcs state machine
ipmi_si: Adding ACPI-specified smic state machine
ipmi_si: Trying SMBIOS-specified kcs state machine at i/o address 0xca8, slave address 0x20, irq 0
ipmi_si: Interface detection failed
ipmi_si: Trying ACPI-specified smic state machine at mem address 0x0, slave address 0x0, irq 0
Could not set up I/O space
ipmi device interface

From reading the PDF manual it states that the IPMI KCS interface is at 0xCA2 in memory, not 0xCA8 that the kernel is trying to probe. Looking at the output from dmidecode shows where this value is probably coming from:

# dmidecode --type 38
# dmidecode 2.11
SMBIOS 2.6 present.
 
Handle 0x001B, DMI type 38, 18 bytes
IPMI Device Information
	Interface Type: KCS (Keyboard Control Style)
	Specification Version: 1.5
	I2C Slave Address: 0x10
	NV Storage Device: Not Present
	Base Address: 0x0000000000000CA8 (I/O)
	Register Spacing: Successive Byte Boundaries

This suggests a minor bug in the BIOS.

Querying the ipmi_si module with modinfo shows it can be persuaded to use a different I/O address so I created /etc/modprobe.d/ipmi.conf containing the following:
1
options ipmi_si type=kcs ports=0xca2
options ipmi_si type=kcs ports=0xca2
Then bounce the service to reload the modules and try again:
# service ipmi restart
Stopping all ipmi drivers:                                 [  OK  ]
Starting ipmi drivers:                                     [  OK  ]
# service ipmi restart Stopping all ipmi drivers: [ OK ] Starting ipmi drivers: [ OK ]

As of CentOS 6.4, the above won’t work as the ipmi_si module is now compiled into the kernel. Instead, you need to edit /etc/grub.conf and append the following to your kernel parameters:

1	ipmi_si.type=kcs ipmi_si.ports=0xca2

Thanks to this post for the info. Instead of bouncing the service you’ll need to reboot, then try again:

# ipmitool chassis status
System Power         : on
Power Overload       : false
Power Interlock      : inactive
Main Power Fault     : false
Power Control Fault  : false
Power Restore Policy : always-off
Last Power Event     : 
Chassis Intrusion    : inactive
Front-Panel Lockout  : inactive
Drive Fault          : false
Cooling/Fan Fault    : false
# ipmitool sdr
Watchdog         | 0x00              | ok
CPU_THEMAL       | 32 degrees C      | ok
NB_THERMAL       | 35 degrees C      | ok
SEL Rate         | 0 messages        | ok
AMBIENT_THERMAL  | 20 degrees C      | ok
EvtLogDisabled   | 0x00              | ok
System Event     | 0x00              | ok
SYS_FAN          | 1000 RPM          | ok
CPU Thermtrip    | 0x00              | ok
Sys Pwr Monitor  | 0x00              | ok

Success! With that sorted, you can now use ipmitool to further configure the management card, although not all of the settings are accessible such as IPv6 network settings so you have to use the BIOS or web interface for some of it.

Overall, I’m fairly happy with the management card. It has decent IPv6 support and the Java KVM client works okay on OS X should I ever need it but I couldn’t coax the separate virtual media client to work, I guess only Windows is supported.

Tags: HP, IPMI, Management, Microserver, OpenIPMI, Remote
Posted in CentOS | 13 Comments »

OpenBSD PPPoE and RFC 4638

January 20th, 2012

I upgraded my Internet connection from ADSL 2+ to FTTC a while ago. I’m with Eclipse as an ISP, but it’s basically the same product as BT Infinity, right down to the Openreach-branded modem, (a Huawei Echolife HG612 to be exact).

With this modem, you need to use a router or some software that can do RFC 2516 PPPoE so I simply kept using OpenBSD on my trusty Soekris net4501 and set up a pppoe(4) interface, job done. However what became apparent is the 133 MHz AMD Elan CPU couldn’t fully utilise the 40 Mb/s bandwidth I now had, at best I could get 16-20 Mb/s with a favourable wind. An upgrade was needed.

Given I’d had around 8 years of flawless service from the net4501, another Soekris board was the way to go. Enter the net6501 with comparatively loads more CPU grunt, RAM and interestingly Gigabit NIC chips; not necessarily for the faster speed, but because they can naturally handle a larger MTU.

The reason for this was that I had read that the Huawei modem and BT FTTC network fully supported RFC 4638, which means you can have an MTU of 1,500 bytes on your PPPoE connection which matches what you’ll have on your internal network. Traditionally a PPPoE connection only allowed 1,492 bytes on account of the overhead of 8 bytes of PPPoE headers in every Ethernet frame payload. Because of this it was almost mandatory to perform MSS clamping on traffic to prevent problems. So having an MTU of 1,500 bytes should avoid the need for any clamping trickery, but means your Ethernet interface needs to cope with an MTU of 1,508 bytes, hence the Gigabit NIC (which can accommodate an MTU of 9,000 bytes with no problems).

One small problem remained, pppoe(4) on OpenBSD 5.0 didn’t support RFC 4638. While I sat down and started to add support I noticed someone had added this to the NetBSD driver already, (which is where the OpenBSD driver originated from), so based on their changes I created a similar patch and with some necessary improvements based on feedback from OpenBSD developers it has now been committed to CVS in time for the 5.1 release.

To make use of the larger MTU is fairly obvious, simply set the MTU explicitly on both the Ethernet and PPPoE interfaces to 8 bytes higher than their default. As an example, my /etc/hostname.em0 now contains:

1	mtu 1508 up

And similarly my /etc/hostname.pppoe0 contains:

inet 0.0.0.0 255.255.255.255 NONE mtu 1500 \
        pppoedev em0 authproto chap \
        authname 'user' authkey 'secret' up
dest 0.0.0.1
!/sbin/route add default -ifp \$if 0.0.0.1

I also added support to tcpdump(8) to display the additional PPPoE tag used to negotiate the larger MTU, so when you bring the interface up, watch for PPP-Max-Payload tags going back and forth during the discovery phase.

With that done the remaining step is to remove any scrub max-mss rules in pf.conf(5) as with any luck they should no longer be required.

Tags: BT Infinity, FTTC, Huawei HG612, MTU, net4501, net6501, PPPoE, rfc 2516, rfc 4638, Soekris
Posted in Networks, OpenBSD | 15 Comments »

PPPoE fixes for natpmpd

December 16th, 2011

I recently started using the pppoe(4) driver on OpenBSD, and with it found a few small bugs in how natpmpd handles these sorts of dynamic interfaces.

One simple bug being it refused to start up if the interface didn’t already exist and also it considered 0.0.0.0 as a valid IP address and would broadcast that to any clients on the network. This situation happened due to the way pppoe(4) interfaces are initially set up and would correct itself quickly once the PPPoE session was negotiated.

Both of these bugs should be fixed so natpmpd should now correctly deny any request until the interface gets created and negotiates a normal IP address, (normal Ethernet interface behaviour should be unchanged).

Tags: NAT-PMP
Posted in OpenBSD | No Comments »

Forcing GUID Partition Table on a disk with CentOS 6

August 6th, 2011

CentOS 6 is now out so I can finally build up a new HP ProLiant Microserver that I purchased for a new home server. Amongst many new features, CentOS 6 ships a GRUB bootloader that can boot from a disk with a GUID Partition Table (GPT for short). Despite not having EFI, the HP BIOS can boot from GPT disks so there’s no limitation with disk sizes that I can use.

However, before I tore down my current home server I wanted to test all of this really did work so I popped a “small” 500 GB disk in the HP. The trouble is the CentOS installer won’t use GPT if the disk is smaller than 2 TB. Normally this wouldn’t be a problem with a single disk apart from I want to specifically test GPT functionality but this can cause complications if your disk is in fact a hardware RAID volume because most RAID controllers allow the following scenario:

You have a RAID 5 volume comprising 4x 500 GB disks, so the OS sees ~ 1.5 TB raw space.
Pull one 500 GB disk, replace with a 1 TB disk, wait for RAID volume to rebuild.
Rinse and repeat until all disks are upgraded.
Grow RAID volume to use the extra space, OS now sees ~ 3 TB raw space.
Resize partitions on the volume, grow filesystems, etc.

You’ll find your volume will have a traditional MBR partition scheme as it fell below the 2 TB limit so you can’t resize the partitions to fully use the new space. While it might be possible to non-destructively convert MBR to GPT, I wouldn’t want to risk it with terabytes of data. It would be far better to just use GPT from the start to save painting myself into a corner later.

If you’re doing a normal install, start the installer until you get to what’s known as stage 2 such that on (ctrl+)alt+F2 you have a working shell. From here, you can use parted to create an empty GPT on the disk:

# /usr/sbin/parted -s /dev/sda mklabel gpt

Return to the installer and keep going until you reach the partitioning choices. Pick the “use empty space” option and the installer will create new partitions within the new GPT. If you choose the “delete everything” option, the installer will replace the GPT with MBR again.

If like me you’re using kickstart to automate the install process, you can do something similar in your kickstart file with the %pre section, something like the following:

%pre
/usr/sbin/parted -s /dev/sda mklabel gpt
%end

Then make sure your kickstart contains no clearpart instructions so it will default to just using the empty space. The only small nit I found after the install is that the minimal install option only includes fdisk and not parted as well so if you want to manage the disk partitions you’ll need to add that either at install time or afterwards as fdisk doesn’t support GPT.

Tags: CentOS 6, GPT, HP, Kickstart, Microserver
Posted in CentOS | 9 Comments »

Storage controller probe order & Kickstart

June 14th, 2011

I’ve been configuring some Dell servers today that have one of Dell’s MD3220 storage arrays attached. The servers were completely blank so they needed kickstarting with CentOS 5.x. Not a problem, just use the standard one that’s used for 99% of the servers.

The problem that arises is the standard kickstart assumes the first disk it sees should be the disks internal to the server and partitions accordingly but annoyingly the installer loads the mpt2sas driver first for the HBA’s used to hook up the storage array, and then loads the megaraid_sas driver for the built-in RAID controller used for the internal disks. This means the internal disks could be anywhere from /dev/sdc onwards depending on what state the array is in.

An undocumented boot option is driverload= which seems to force the supplied list of drivers to be loaded first before anything else that’s detected. I found reference to this option in this post along with use of the nostorage boot option. However when I used the following options:

nostorage driverload=sd_mod:megaraid_sas

The installer still loaded the mpt2sas driver but it did force the megaraid_sas driver to be loaded first. This could just be a bug that mpt2sas isn’t on the list of “storage” drivers but the net result is acceptable in that /dev/sda now pointed to the internal disks. After the server rebooted /etc/modprobe.conf also had the correct scsi_hostadapterX aliases to keep things in the right order.

Tags: CentOS 5, Kickstart
Posted in CentOS | No Comments »