Archive for the ‘CentOS’ Category

HP Remote Management card and newer versions of Java

Thursday, October 17th, 2019

I was having some problems with DKMS and the ZFS kernel modules on my HP Microserver and I figured when it came to the inevitable reboot, this would be when the OS decides to kernel panic because the modules weren’t built properly and not boot back up again.

“No problem, I have the Remote Management card, I’ll just connect to that and watch it boot to make sure everything works.”, I thought. So once I remembered the password and logged in, clicked to launch the KVM Viewer, then discovered this Mac doesn’t have Java installed, (and I’ve managed for this long without it), go off to the Oracle website, relinquish all rights to my soul, and finally download and install Java 8.

I then retried launching the KVM Viewer and was met with a “Connection Failed” error. Now, this definitely used to work, so I went searching for clues. Nobody else had reported problems specifically with the Microserver card, but plenty were having problems with the iLO cards that are fitted to the Microserver’s bigger siblings, and also the iDRAC cards fitted to Dell servers; they all use pretty much the same Avocent-based Java client.

Some people suggested downgrading to Java 7 would get things working however Oracle have pulled all downloads of Java 7 unless you also promise them your first-born. Then I got the clue that it might simply be that the client is trying to use some crypto algorithm that has since been disabled in newer Java versions. Given the last firmware release for the management card was released in 2014, a lot has changed in the crypto landscape since then.

The security policy for Java is configured in a file named java.security which for the default Java install on a Mac, lives under /Library/Internet Plug-Ins/JavaAppletPlugin.plugin/Contents/Home/lib/security/. In that file, I found the following lines:

jdk.tls.disabledAlgorithms=SSLv3, RC4, DES, MD5withRSA, DH keySize < 1024, \
    EC keySize < 224, 3DES_EDE_CBC, anon, NULL

So for a first attempt, I commented both lines out and retried the KVM Viewer, and bingo, it worked again. Through a process of elimination, I figured out that it’s the 3DES_EDE_CBC algorithm that needs to be re-enabled. So here is that pro-tip for anyone else that is still using one of these, you should amend the line to:

jdk.tls.disabledAlgorithms=SSLv3, RC4, DES, MD5withRSA, DH keySize < 1024, \
    EC keySize < 224, anon, NULL

And in the end after all of that faffing, the Microserver rebooted without a hiccup, no kernel panic. Typical.

Monitoring file modifications with auditd with exceptions

Wednesday, November 19th, 2014

Playing with auditd, I had a need to monitor file modifications for all files recursively underneath a given directory. According to the auditctl(8) man page there are two ways of writing a rule to do this:

-w /directory/ -p wa
-a exit,always -F dir=/directory/ -F perm=wa

The former rule is basically a shortcut for the latter rule; the latter rule is also potentially more expressive with the addition of extra -F conditions. I also needed to ideally exclude certain files and/or sub-directories in the directory from triggering the audit rule and it turns out you do this:

-a exit,never -F dir=/directory/directory-to-exclude/
-a exit,never -F path=/directory/file-to-exclude
-a exit,always -F dir=/directory/ -F perm=wa

According to this post order is important; list the exceptions before the main rule.

Building mod_proxy_wstunnel for CentOS 6

Wednesday, August 6th, 2014

I had a need to be able to put an Apache-based reverse proxy in front of an install of Uchiwa which is a Node.js-based dashboard for Sensu. The only problem is that it uses WebSockets which means it doesn’t work with the regular mod_proxy_http module. In version 2.4.5 onwards there is mod_proxy_wstunnel which fills in the gap however CentOS 6 only has a 2.2.15 (albeit heavily patched) package.

There are various instructions on how to backport the module for 2.2.x (mostly for Ubuntu) but these involve compiling the whole of Apache from source again with the module added via an additional patch. I don’t want to maintain my own Apache packages but more importantly Apache has provided apxs a.k.a the APache eXtenSion tool to compile external modules without requiring the whole source tree available.

So, I have created a standalone RPM package for CentOS 6 that just installs the mod_proxy_wstunnel module alongside the standard httpd RPM package. In order to do this I took the original patch and removed the alterations to the various build files and also flattened the source into a single file, (the code changes were basically adding whole new functions so they were fine to just inline together). The revised source file and accompanying RPM spec file are available in this Github gist.

HP Microserver TPM

Friday, January 10th, 2014

I’ve been dabbling with DNSSEC which involves creating a few zone- and key-signing keys, and it became immediately apparent that my headless HP Microserver has very poor entropy generation for /dev/random. After poking and prodding it became apparent there’s no dormant hardware RNG that I can just enable to fix it.

Eventually I stumbled on this post which suggests you can install and make use of the optional TPM as a source of entropy.

I picked up one cheaply and installed it following the above instructions to install and configure it; I found I only needed to remove the power cord for safety’s sake, the TPM connector on the motherboard is right at the front so I didn’t need to pull the tray out.

Also, since that blog post, the rng-tools package on RHEL/CentOS 6.x now includes an init script so it’s just a case of doing the following final step:

# chkconfig rngd on
# service rngd start

It should then be possible to pass this up to any KVM guests using the virtio-rng.ko module.

HP Microserver Remote Management Card

Saturday, January 28th, 2012

I recently acquired the Remote Management card for my HP Microserver, which allows remote KVM & power control, IPMI management and hardware monitoring through temperature & fan sensors.

Note the extra connector on the card in addition to the standard PCI-e x1 connector which matches the dedicated slot on the Microserver motherboard. This presented a bit of a problem as I was using the space for the battery backup module for the RAID controller in the neighbouring slot.

Thankfully the long ribbon cable meant I could route the battery up to the space behind the DVD burner freeing the slot again. Once the card was installed and everything screwed back together I booted straight back into CentOS. Given IPMI is touted as a feature I figured that was the first thing to try so I installed OpenIPMI:

# yum -y install OpenIPMI ipmitool
...
# service ipmi start
Starting ipmi drivers:                                     [  OK  ]
# ipmitool chassis status
Could not open device at /dev/ipmi0 or /dev/ipmi/0 or /dev/ipmidev/0: No such file or directory
Error sending Chassis Status command

Hmm, not good. Looking at dmesg shows the following is output when the IPMI drivers get loaded:

ipmi message handler version 39.2
IPMI System Interface driver.
ipmi_si: Adding SMBIOS-specified kcs state machine
ipmi_si: Adding ACPI-specified smic state machine
ipmi_si: Trying SMBIOS-specified kcs state machine at i/o address 0xca8, slave address 0x20, irq 0
ipmi_si: Interface detection failed
ipmi_si: Trying ACPI-specified smic state machine at mem address 0x0, slave address 0x0, irq 0
Could not set up I/O space
ipmi device interface

From reading the PDF manual it states that the IPMI KCS interface is at 0xCA2 in memory, not 0xCA8 that the kernel is trying to probe. Looking at the output from dmidecode shows where this value is probably coming from:

# dmidecode --type 38
# dmidecode 2.11
SMBIOS 2.6 present.
 
Handle 0x001B, DMI type 38, 18 bytes
IPMI Device Information
	Interface Type: KCS (Keyboard Control Style)
	Specification Version: 1.5
	I2C Slave Address: 0x10
	NV Storage Device: Not Present
	Base Address: 0x0000000000000CA8 (I/O)
	Register Spacing: Successive Byte Boundaries

This suggests a minor bug in the BIOS.

Querying the ipmi_si module with modinfo shows it can be persuaded to use a different I/O address so I created /etc/modprobe.d/ipmi.conf containing the following:

1
options ipmi_si type=kcs ports=0xca2

Then bounce the service to reload the modules and try again:

# service ipmi restart
Stopping all ipmi drivers:                                 [  OK  ]
Starting ipmi drivers:                                     [  OK  ]

As of CentOS 6.4, the above won’t work as the ipmi_si module is now compiled into the kernel. Instead, you need to edit /etc/grub.conf and append the following to your kernel parameters:

1
ipmi_si.type=kcs ipmi_si.ports=0xca2

Thanks to this post for the info. Instead of bouncing the service you’ll need to reboot, then try again:

# ipmitool chassis status
System Power         : on
Power Overload       : false
Power Interlock      : inactive
Main Power Fault     : false
Power Control Fault  : false
Power Restore Policy : always-off
Last Power Event     : 
Chassis Intrusion    : inactive
Front-Panel Lockout  : inactive
Drive Fault          : false
Cooling/Fan Fault    : false
# ipmitool sdr
Watchdog         | 0x00              | ok
CPU_THEMAL       | 32 degrees C      | ok
NB_THERMAL       | 35 degrees C      | ok
SEL Rate         | 0 messages        | ok
AMBIENT_THERMAL  | 20 degrees C      | ok
EvtLogDisabled   | 0x00              | ok
System Event     | 0x00              | ok
SYS_FAN          | 1000 RPM          | ok
CPU Thermtrip    | 0x00              | ok
Sys Pwr Monitor  | 0x00              | ok

Success! With that sorted, you can now use ipmitool to further configure the management card, although not all of the settings are accessible such as IPv6 network settings so you have to use the BIOS or web interface for some of it.

Overall, I’m fairly happy with the management card. It has decent IPv6 support and the Java KVM client works okay on OS X should I ever need it but I couldn’t coax the separate virtual media client to work, I guess only Windows is supported.

Forcing GUID Partition Table on a disk with CentOS 6

Saturday, August 6th, 2011

CentOS 6 is now out so I can finally build up a new HP ProLiant Microserver that I purchased for a new home server. Amongst many new features, CentOS 6 ships a GRUB bootloader that can boot from a disk with a GUID Partition Table (GPT for short). Despite not having EFI, the HP BIOS can boot from GPT disks so there’s no limitation with disk sizes that I can use.

However, before I tore down my current home server I wanted to test all of this really did work so I popped a “small” 500 GB disk in the HP. The trouble is the CentOS installer won’t use GPT if the disk is smaller than 2 TB. Normally this wouldn’t be a problem with a single disk apart from I want to specifically test GPT functionality but this can cause complications if your disk is in fact a hardware RAID volume because most RAID controllers allow the following scenario:

  1. You have a RAID 5 volume comprising 4x 500 GB disks, so the OS sees ~ 1.5 TB raw space.
  2. Pull one 500 GB disk, replace with a 1 TB disk, wait for RAID volume to rebuild.
  3. Rinse and repeat until all disks are upgraded.
  4. Grow RAID volume to use the extra space, OS now sees ~ 3 TB raw space.
  5. Resize partitions on the volume, grow filesystems, etc.

You’ll find your volume will have a traditional MBR partition scheme as it fell below the 2 TB limit so you can’t resize the partitions to fully use the new space. While it might be possible to non-destructively convert MBR to GPT, I wouldn’t want to risk it with terabytes of data. It would be far better to just use GPT from the start to save painting myself into a corner later.

If you’re doing a normal install, start the installer until you get to what’s known as stage 2 such that on (ctrl+)alt+F2 you have a working shell. From here, you can use parted to create an empty GPT on the disk:

# /usr/sbin/parted -s /dev/sda mklabel gpt

Return to the installer and keep going until you reach the partitioning choices. Pick the “use empty space” option and the installer will create new partitions within the new GPT. If you choose the “delete everything” option, the installer will replace the GPT with MBR again.

If like me you’re using kickstart to automate the install process, you can do something similar in your kickstart file with the %pre section, something like the following:

%pre
/usr/sbin/parted -s /dev/sda mklabel gpt
%end

Then make sure your kickstart contains no clearpart instructions so it will default to just using the empty space. The only small nit I found after the install is that the minimal install option only includes fdisk and not parted as well so if you want to manage the disk partitions you’ll need to add that either at install time or afterwards as fdisk doesn’t support GPT.

Storage controller probe order & Kickstart

Tuesday, June 14th, 2011

I’ve been configuring some Dell servers today that have one of Dell’s MD3220 storage arrays attached. The servers were completely blank so they needed kickstarting with CentOS 5.x. Not a problem, just use the standard one that’s used for 99% of the servers.

The problem that arises is the standard kickstart assumes the first disk it sees should be the disks internal to the server and partitions accordingly but annoyingly the installer loads the mpt2sas driver first for the HBA’s used to hook up the storage array, and then loads the megaraid_sas driver for the built-in RAID controller used for the internal disks. This means the internal disks could be anywhere from /dev/sdc onwards depending on what state the array is in.

An undocumented boot option is driverload= which seems to force the supplied list of drivers to be loaded first before anything else that’s detected. I found reference to this option in this post along with use of the nostorage boot option. However when I used the following options:

nostorage driverload=sd_mod:megaraid_sas

The installer still loaded the mpt2sas driver but it did force the megaraid_sas driver to be loaded first. This could just be a bug that mpt2sas isn’t on the list of “storage” drivers but the net result is acceptable in that /dev/sda now pointed to the internal disks. After the server rebooted /etc/modprobe.conf also had the correct scsi_hostadapterX aliases to keep things in the right order.

Holy Exploding Batteries Batman

Sunday, February 27th, 2011

If you have a UPS, it’s probably worth paying attention to the temperature reading if it reports such a thing. My APC Smart-UPS 1000VA suffered a failed battery which wasn’t totally unexpected given I had last replaced it about two years ago.

While I was awaiting the replacement RBC6 battery, I had a poke about with apcupsd and noticed that it reckoned the internal temperature of the UPS was around 39-40 ℃ which seemed a bit high.

New battery arrived so I pulled the front panel off the UPS and tried to use that stupid plastic flap APC put on the battery to help you pull it out, surprise, it tore off like it’s done on every damn battery. Okay, so I’d have to reach in and get my hand behind the battery to push it out, which due to the nature of what a UPS generally does felt like that bit in the Flash Gordon movie when Peter Duncan sticks his arm in the tree stump and gets fatally stung. Nope, still not coming out and I could feel the battery was quite hot so I pulled the power and left it to cool down for an hour or two. With a bit of help from gravity it became apparent why the battery was being so stubborn:

Bursting Battery

The photo probably doesn’t illustrate just how distorted each side of the battery was. According to apcupsd the temperature is now around 25 ℃ with the new battery so it’s probably worth sticking some monitoring on that. Grabbing the temperature is easy if apcupsd(8) is running:

# /usr/sbin/apcaccess | grep -i temp
ITEMP    : 25.9 C Internal

It should be easy enough to wrap that in a simple Nagios test and/or graph it with Cacti.

This blog is IPv6-enabled

Sunday, December 5th, 2010

I’ve been quite keen on getting IPv6 connectivity to my hosts, although I’m currently lacking either a home internet package or hosted server where there is native connectivity. Thankfully there are enough free tunnel providers available which work fine as a workaround.

I’m using SixXS for two tunnels; one to my home network and another to the VPS provided by Linode which amongst other things hosts this blog.

SixXS operate a system of credits whereby tasks such as creating tunnels, obtaining subnets and setting up reverse DNS delegation costs varying amounts of credits and the only way to earn credits is to keep your tunnel up and running on a week-by-week basis. In the interests of promoting reliable and stable connectivity this is a Good Thing.

Now, what must have happened about a year ago was I created the two tunnels, but because SixXS deliberately don’t give you enough credits to be able to request subnets at the same time, you have to wait a week or two to earn sufficient credits to be able to do that. So I got my home network sorted first as that was more important and then likely got distracted by something shiny and forgot to sort out a subnet for the server.

Now, you don’t necessarily need a subnet when you have just the one host at the end a tunnel, you can just use the IPv6 address allocated to the local endpoint of the tunnel. However, you can’t change the reverse DNS of that address, meaning it will be stuck called something like cl-123.pop-01.us.sixxs.net. If you’re planning on running a service which can be picky about the reverse DNS, (SMTP is a good example), then it’s probably best to get a subnet so you have full control over it.

By this point, requesting a subnet isn’t a problem credits-wise as the tunnels have been up for over a year. My existing tunnel configuration stored under /etc/sysconfig/network-scripts/ifcfg-sit1 looked something like this:

1
2
3
4
5
6
7
8
9
10
DEVICE=sit1
BOOTPROTO=none
ONBOOT=yes
IPV6INIT=yes
IPV6_TUNNELNAME=SixXS
IPV6TUNNELIPV4=1.2.3.4
IPV6TUNNELIPV4LOCAL=5.6.7.8
IPV6ADDR=2001:dead:beef:7a::2/64
IPV6_MTU=1280
TYPE=sit

Once the subnet has been allocated and routed to your tunnel, it’s simply a case of picking an address from that and altering the configuration like so:

1
2
3
4
5
6
7
8
9
10
11
DEVICE=sit1
BOOTPROTO=none
ONBOOT=yes
IPV6INIT=yes
IPV6_TUNNELNAME=SixXS
IPV6TUNNELIPV4=1.2.3.4
IPV6TUNNELIPV4LOCAL=5.6.7.8
IPV6ADDR=2001:dead:beef:7a::2/64
IPV6ADDR_SECONDARIES="2001:feed:beef::1/128"
IPV6_MTU=1280
TYPE=sit

Note the IPV6ADDR_SECONDARIES addition. Now your host should be accessible through both addresses and thanks to #199862 any outbound connections initiated from your host should use the subnet address rather than the tunnel address. You can test this with ip(8) and some known IPv6-connected hosts.

# host -t aaaa www.sixxs.net
www.sixxs.net is an alias for nginx.sixxs.net.
nginx.sixxs.net has IPv6 address 2001:1af8:1:f006::6
nginx.sixxs.net has IPv6 address 2001:838:2:1::30:67
nginx.sixxs.net has IPv6 address 2001:960:800::2
# ip -6 route get 2001:1af8:1:f006::6
2001:1af8:1:f006::6 via 2001:1af8:1:f006::6 dev sit1  src 2001:feed:beef::1  metric 0 
    cache  mtu 1280 advmss 1220 hoplimit 4294967295
# host -t aaaa ipv6.google.com
ipv6.google.com is an alias for ipv6.l.google.com.
ipv6.l.google.com has IPv6 address 2001:4860:800f::63
# ip -6 route get 2001:4860:800f::63
2001:4860:800f::63 via 2001:4860:800f::63 dev sit1  src 2001:feed:beef::1  metric 0 
    cache  mtu 1280 advmss 1220 hoplimit 4294967295

Trying again for the remote endpoint address of the tunnel gets a different result:

# ip -6 route get 2001:dead:beef:7a::1
2001:dead:beef:7a::1 via :: dev sit1  src 2001:dead:beef:7a::2  metric 256  expires 21315606sec mtu 1280 advmss 1220 hoplimit 4294967295

The kernel decides that in this case using the local address of the tunnel is a better choice, which I think is due to RFC 3484 and how Linux chooses a source address. If that expiry counter ever hits zero and the behaviour changes, I’ll let you know…

Issues with iptables stateful filtering

Wednesday, September 29th, 2010

I hit a weird issue today, I have Apache configured as a reverse proxy using mod_proxy_balancer which is a welcome addition in Apache 2.2.x. This is forwarding selected requests to more Apache instances running mod_perl, although this could be any sort of application layer, pretty standard stuff.

With ProxyStatus enabled and pushing a reasonable amount of traffic through the proxy, I started to notice that the application layer Apache instances would consistently get marked in error state every so often, removing them from the pool of available servers until their cooldown period expired and the proxy enabled them again.

Investigating the logs showed up this error:

[Tue Sep 28 00:24:39 2010] [error] (113)No route to host: proxy: HTTP: attempt to connect to 192.0.2.1:80 (192.0.2.1) failed
[Tue Sep 28 00:24:39 2010] [error] ap_proxy_connect_backend disabling worker for (192.0.2.1)

A spot of the ol’ google-fu turned up descriptions of similar problems, some not even related to Apache. The problem looked related to iptables.

All the servers are running CentOS 5 and anyone who runs this is probably aware of the stock Red Hat iptables ruleset. With HTTP access enabled, it looks something similar to this:

1
2
3
4
5
6
7
8
9
10
11
12
13
-A INPUT -j RH-Firewall-1-INPUT
-A FORWARD -j RH-Firewall-1-INPUT
-A RH-Firewall-1-INPUT -i lo -j ACCEPT
-A RH-Firewall-1-INPUT -p icmp --icmp-type any -j ACCEPT
-A RH-Firewall-1-INPUT -p 50 -j ACCEPT
-A RH-Firewall-1-INPUT -p 51 -j ACCEPT
-A RH-Firewall-1-INPUT -p udp --dport 5353 -d 224.0.0.251 -j ACCEPT
-A RH-Firewall-1-INPUT -p udp -m udp --dport 631 -j ACCEPT
-A RH-Firewall-1-INPUT -p tcp -m tcp --dport 631 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 22 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 80 -j ACCEPT
-A RH-Firewall-1-INPUT -j REJECT --reject-with icmp-host-prohibited

Almost all of it is boilerplate apart from line 12, which I added and is identical to the line above it granting access to SSH. Analysing the live hit counts against each of these rules showed a large number hitting that last catch-all rule on line 13 and indeed, this is what is causing the Apache errors.

Analysing the traffic with tcpdump/wireshark showed that the frontend Apache server is only getting as far as sending the initial SYN packet and it’s failing to match either the dedicated rule on line 12 for HTTP traffic, or even the rule on line 10 to match any related or previously established traffic, although I wouldn’t really expect it to match that.

Adding a rule before the last one to match and log any HTTP packets that are considered to be in the state INVALID showed that indeed for some strange reason, iptables is deciding that an initial SYN is somehow invalid.

More information about why it might be invalid can be coaxed from the kernel by issuing the following:

# echo 255 > /proc/sys/net/ipv4/netfilter/ip_conntrack_log_invalid

Although all this gave me was some extra text saying “invalid state” and the same output you get from the standard logging target:

ip_ct_tcp: invalid state IN= OUT= SRC=192.0.2.2 DST=192.0.2.1 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=21158 DF PROTO=TCP SPT=57351 DPT=80 SEQ=1402246598 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405B40402080A291DC7600000000001030307)

From searching the netfilter bugzilla for any matching bug reports I found a knob that relaxes the connection tracking, enabled with the following:

# echo 1 > /proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_be_liberal

This didn’t fix things, plus it’s recommended to only use this in extreme cases with a broken router or firewall. As all of these machines are on the same network segment with just a switch connecting them there shouldn’t be anything mangling the packets.

With those avenues exhausted the suggested workaround of adding rules similar to the following:

1
-A RH-Firewall-1-INPUT -m tcp -p tcp --dport 80 --syn -j REJECT --reject-with tcp-reset

before your last rule doesn’t really work as the proxy still drops the servers from the pool as before, but just logs a different reason instead:

[Tue Sep 28 13:59:23 2010] [error] (111)Connection refused: proxy: HTTP: attempt to connect to 192.0.2.1:80 (192.0.2.1) failed
[Tue Sep 28 13:59:23 2010] [error] ap_proxy_connect_backend disabling worker for (192.0.2.1)

This isn’t really acceptable to me anyway as it requires your proxy to retry in response to a server politely telling it to go away and it’s going to be a performance hit. The whole point of using mod_proxy_balancer for me was so it could legitimately remove servers that are dead or administratively stopped, how is it supposed to tell the difference between that and a dodgy firewall?

The only solution that worked and was deemed acceptable was to simply remove the state requirement on matching the HTTP traffic, like so:

-A RH-Firewall-1-INPUT -m tcp -p tcp --dport 80 -j ACCEPT

This will match both new and invalid packets, however it’s left me with a pretty low opinion of iptables now. Give me pf any day.