Archive for the ‘CentOS’ Category

Altera USB Blaster udev rules

Wednesday, October 14th, 2020

Just an aide-mémoire for how I got my Altera USB Blaster to work as a non-root user on CentOS Linux for use with with Quartus. Create /etc/udev/rules.d/51-usbblaster.rules with the following:

# Intel FPGA Download Cable
SUBSYSTEM=="usb", ATTR{idVendor}=="09fb", ATTR{idProduct}=="6001", MODE="0666"
SUBSYSTEM=="usb", ATTR{idVendor}=="09fb", ATTR{idProduct}=="6002", MODE="0666"
SUBSYSTEM=="usb", ATTR{idVendor}=="09fb", ATTR{idProduct}=="6003", MODE="0666"

# Intel FPGA Download Cable II
SUBSYSTEM=="usb", ATTR{idVendor}=="09fb", ATTR{idProduct}=="6010", MODE="0666"
SUBSYSTEM=="usb", ATTR{idVendor}=="09fb", ATTR{idProduct}=="6810", MODE="0666"

Run the following to reload the rules:

# udevadm control --reload

Find where the device is:

# lsusb
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 002 Device 002: ID 09fb:6001 Altera Blaster
Bus 002 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub

Confirm the device node now has the correct mode:

# ls -l /dev/bus/usb/002/002 
crw-rw-rw-. 1 root root 189, 129 Oct 14 11:56 /dev/bus/usb/002/002

HP Remote Management card and newer versions of Java

Thursday, October 17th, 2019

I was having some problems with DKMS and the ZFS kernel modules on my HP Microserver and I figured when it came to the inevitable reboot, this would be when the OS decides to kernel panic because the modules weren’t built properly and not boot back up again.

“No problem, I have the Remote Management card, I’ll just connect to that and watch it boot to make sure everything works.”, I thought. So once I remembered the password and logged in, clicked to launch the KVM Viewer, then discovered this Mac doesn’t have Java installed, (and I’ve managed for this long without it), go off to the Oracle website, relinquish all rights to my soul, and finally download and install Java 8.

I then retried launching the KVM Viewer and was met with a “Connection Failed” error. Now, this definitely used to work, so I went searching for clues. Nobody else had reported problems specifically with the Microserver card, but plenty were having problems with the iLO cards that are fitted to the Microserver’s bigger siblings, and also the iDRAC cards fitted to Dell servers; they all use pretty much the same Avocent-based Java client.

Some people suggested downgrading to Java 7 would get things working however Oracle have pulled all downloads of Java 7 unless you also promise them your first-born. Then I got the clue that it might simply be that the client is trying to use some crypto algorithm that has since been disabled in newer Java versions. Given the last firmware release for the management card was released in 2014, a lot has changed in the crypto landscape since then.

The security policy for Java is configured in a file named java.security which for the default Java install on a Mac, lives under /Library/Internet Plug-Ins/JavaAppletPlugin.plugin/Contents/Home/lib/security/. In that file, I found the following lines:

jdk.tls.disabledAlgorithms=SSLv3, RC4, DES, MD5withRSA, DH keySize < 1024, \
    EC keySize < 224, 3DES_EDE_CBC, anon, NULL

So for a first attempt, I commented both lines out and retried the KVM Viewer, and bingo, it worked again. Through a process of elimination, I figured out that it’s the 3DES_EDE_CBC algorithm that needs to be re-enabled. So here is that pro-tip for anyone else that is still using one of these, you should amend the line to:

jdk.tls.disabledAlgorithms=SSLv3, RC4, DES, MD5withRSA, DH keySize < 1024, \
    EC keySize < 224, anon, NULL

And in the end after all of that faffing, the Microserver rebooted without a hiccup, no kernel panic. Typical.

Monitoring file modifications with auditd with exceptions

Wednesday, November 19th, 2014

Playing with auditd, I had a need to monitor file modifications for all files recursively underneath a given directory. According to the auditctl(8) man page there are two ways of writing a rule to do this:

-w /directory/ -p wa
-a exit,always -F dir=/directory/ -F perm=wa

The former rule is basically a shortcut for the latter rule; the latter rule is also potentially more expressive with the addition of extra -F conditions. I also needed to ideally exclude certain files and/or sub-directories in the directory from triggering the audit rule and it turns out you do this:

-a exit,never -F dir=/directory/directory-to-exclude/
-a exit,never -F path=/directory/file-to-exclude
-a exit,always -F dir=/directory/ -F perm=wa

According to this post order is important; list the exceptions before the main rule.

Building mod_proxy_wstunnel for CentOS 6

Wednesday, August 6th, 2014

I had a need to be able to put an Apache-based reverse proxy in front of an install of Uchiwa which is a Node.js-based dashboard for Sensu. The only problem is that it uses WebSockets which means it doesn’t work with the regular mod_proxy_http module. In version 2.4.5 onwards there is mod_proxy_wstunnel which fills in the gap however CentOS 6 only has a 2.2.15 (albeit heavily patched) package.

There are various instructions on how to backport the module for 2.2.x (mostly for Ubuntu) but these involve compiling the whole of Apache from source again with the module added via an additional patch. I don’t want to maintain my own Apache packages but more importantly Apache has provided apxs a.k.a the APache eXtenSion tool to compile external modules without requiring the whole source tree available.

So, I have created a standalone RPM package for CentOS 6 that just installs the mod_proxy_wstunnel module alongside the standard httpd RPM package. In order to do this I took the original patch and removed the alterations to the various build files and also flattened the source into a single file, (the code changes were basically adding whole new functions so they were fine to just inline together). The revised source file and accompanying RPM spec file are available in this Github gist.

HP Microserver TPM

Friday, January 10th, 2014

I’ve been dabbling with DNSSEC which involves creating a few zone- and key-signing keys, and it became immediately apparent that my headless HP Microserver has very poor entropy generation for /dev/random. After poking and prodding it became apparent there’s no dormant hardware RNG that I can just enable to fix it.

Eventually I stumbled on this post which suggests you can install and make use of the optional TPM as a source of entropy.

I picked up one cheaply and installed it following the above instructions to install and configure it; I found I only needed to remove the power cord for safety’s sake, the TPM connector on the motherboard is right at the front so I didn’t need to pull the tray out.

Also, since that blog post, the rng-tools package on RHEL/CentOS 6.x now includes an init script so it’s just a case of doing the following final step:

# chkconfig rngd on
# service rngd start

It should then be possible to pass this up to any KVM guests using the virtio-rng.ko module.

HP Microserver Remote Management Card

Saturday, January 28th, 2012

I recently acquired the Remote Management card for my HP Microserver, which allows remote KVM & power control, IPMI management and hardware monitoring through temperature & fan sensors.

Note the extra connector on the card in addition to the standard PCI-e x1 connector which matches the dedicated slot on the Microserver motherboard. This presented a bit of a problem as I was using the space for the battery backup module for the RAID controller in the neighbouring slot.

Thankfully the long ribbon cable meant I could route the battery up to the space behind the DVD burner freeing the slot again. Once the card was installed and everything screwed back together I booted straight back into CentOS. Given IPMI is touted as a feature I figured that was the first thing to try so I installed OpenIPMI:

# yum -y install OpenIPMI ipmitool
...
# service ipmi start
Starting ipmi drivers:                                     [  OK  ]
# ipmitool chassis status
Could not open device at /dev/ipmi0 or /dev/ipmi/0 or /dev/ipmidev/0: No such file or directory
Error sending Chassis Status command

Hmm, not good. Looking at dmesg shows the following is output when the IPMI drivers get loaded:

ipmi message handler version 39.2
IPMI System Interface driver.
ipmi_si: Adding SMBIOS-specified kcs state machine
ipmi_si: Adding ACPI-specified smic state machine
ipmi_si: Trying SMBIOS-specified kcs state machine at i/o address 0xca8, slave address 0x20, irq 0
ipmi_si: Interface detection failed
ipmi_si: Trying ACPI-specified smic state machine at mem address 0x0, slave address 0x0, irq 0
Could not set up I/O space
ipmi device interface

From reading the PDF manual it states that the IPMI KCS interface is at 0xCA2 in memory, not 0xCA8 that the kernel is trying to probe. Looking at the output from dmidecode shows where this value is probably coming from:

# dmidecode --type 38
# dmidecode 2.11
SMBIOS 2.6 present.
 
Handle 0x001B, DMI type 38, 18 bytes
IPMI Device Information
	Interface Type: KCS (Keyboard Control Style)
	Specification Version: 1.5
	I2C Slave Address: 0x10
	NV Storage Device: Not Present
	Base Address: 0x0000000000000CA8 (I/O)
	Register Spacing: Successive Byte Boundaries

This suggests a minor bug in the BIOS.

Querying the ipmi_si module with modinfo shows it can be persuaded to use a different I/O address so I created /etc/modprobe.d/ipmi.conf containing the following:

1
options ipmi_si type=kcs ports=0xca2

Then bounce the service to reload the modules and try again:

# service ipmi restart
Stopping all ipmi drivers:                                 [  OK  ]
Starting ipmi drivers:                                     [  OK  ]

As of CentOS 6.4, the above won’t work as the ipmi_si module is now compiled into the kernel. Instead, you need to edit /etc/grub.conf and append the following to your kernel parameters:

1
ipmi_si.type=kcs ipmi_si.ports=0xca2

Thanks to this post for the info. Instead of bouncing the service you’ll need to reboot, then try again:

# ipmitool chassis status
System Power         : on
Power Overload       : false
Power Interlock      : inactive
Main Power Fault     : false
Power Control Fault  : false
Power Restore Policy : always-off
Last Power Event     : 
Chassis Intrusion    : inactive
Front-Panel Lockout  : inactive
Drive Fault          : false
Cooling/Fan Fault    : false
# ipmitool sdr
Watchdog         | 0x00              | ok
CPU_THEMAL       | 32 degrees C      | ok
NB_THERMAL       | 35 degrees C      | ok
SEL Rate         | 0 messages        | ok
AMBIENT_THERMAL  | 20 degrees C      | ok
EvtLogDisabled   | 0x00              | ok
System Event     | 0x00              | ok
SYS_FAN          | 1000 RPM          | ok
CPU Thermtrip    | 0x00              | ok
Sys Pwr Monitor  | 0x00              | ok

Success! With that sorted, you can now use ipmitool to further configure the management card, although not all of the settings are accessible such as IPv6 network settings so you have to use the BIOS or web interface for some of it.

Overall, I’m fairly happy with the management card. It has decent IPv6 support and the Java KVM client works okay on OS X should I ever need it but I couldn’t coax the separate virtual media client to work, I guess only Windows is supported.

Forcing GUID Partition Table on a disk with CentOS 6

Saturday, August 6th, 2011

CentOS 6 is now out so I can finally build up a new HP ProLiant Microserver that I purchased for a new home server. Amongst many new features, CentOS 6 ships a GRUB bootloader that can boot from a disk with a GUID Partition Table (GPT for short). Despite not having EFI, the HP BIOS can boot from GPT disks so there’s no limitation with disk sizes that I can use.

However, before I tore down my current home server I wanted to test all of this really did work so I popped a “small” 500 GB disk in the HP. The trouble is the CentOS installer won’t use GPT if the disk is smaller than 2 TB. Normally this wouldn’t be a problem with a single disk apart from I want to specifically test GPT functionality but this can cause complications if your disk is in fact a hardware RAID volume because most RAID controllers allow the following scenario:

  1. You have a RAID 5 volume comprising 4x 500 GB disks, so the OS sees ~ 1.5 TB raw space.
  2. Pull one 500 GB disk, replace with a 1 TB disk, wait for RAID volume to rebuild.
  3. Rinse and repeat until all disks are upgraded.
  4. Grow RAID volume to use the extra space, OS now sees ~ 3 TB raw space.
  5. Resize partitions on the volume, grow filesystems, etc.

You’ll find your volume will have a traditional MBR partition scheme as it fell below the 2 TB limit so you can’t resize the partitions to fully use the new space. While it might be possible to non-destructively convert MBR to GPT, I wouldn’t want to risk it with terabytes of data. It would be far better to just use GPT from the start to save painting myself into a corner later.

If you’re doing a normal install, start the installer until you get to what’s known as stage 2 such that on (ctrl+)alt+F2 you have a working shell. From here, you can use parted to create an empty GPT on the disk:

# /usr/sbin/parted -s /dev/sda mklabel gpt

Return to the installer and keep going until you reach the partitioning choices. Pick the “use empty space” option and the installer will create new partitions within the new GPT. If you choose the “delete everything” option, the installer will replace the GPT with MBR again.

If like me you’re using kickstart to automate the install process, you can do something similar in your kickstart file with the %pre section, something like the following:

%pre
/usr/sbin/parted -s /dev/sda mklabel gpt
%end

Then make sure your kickstart contains no clearpart instructions so it will default to just using the empty space. The only small nit I found after the install is that the minimal install option only includes fdisk and not parted as well so if you want to manage the disk partitions you’ll need to add that either at install time or afterwards as fdisk doesn’t support GPT.

Storage controller probe order & Kickstart

Tuesday, June 14th, 2011

I’ve been configuring some Dell servers today that have one of Dell’s MD3220 storage arrays attached. The servers were completely blank so they needed kickstarting with CentOS 5.x. Not a problem, just use the standard one that’s used for 99% of the servers.

The problem that arises is the standard kickstart assumes the first disk it sees should be the disks internal to the server and partitions accordingly but annoyingly the installer loads the mpt2sas driver first for the HBA’s used to hook up the storage array, and then loads the megaraid_sas driver for the built-in RAID controller used for the internal disks. This means the internal disks could be anywhere from /dev/sdc onwards depending on what state the array is in.

An undocumented boot option is driverload= which seems to force the supplied list of drivers to be loaded first before anything else that’s detected. I found reference to this option in this post along with use of the nostorage boot option. However when I used the following options:

nostorage driverload=sd_mod:megaraid_sas

The installer still loaded the mpt2sas driver but it did force the megaraid_sas driver to be loaded first. This could just be a bug that mpt2sas isn’t on the list of “storage” drivers but the net result is acceptable in that /dev/sda now pointed to the internal disks. After the server rebooted /etc/modprobe.conf also had the correct scsi_hostadapterX aliases to keep things in the right order.

Holy Exploding Batteries Batman

Sunday, February 27th, 2011

If you have a UPS, it’s probably worth paying attention to the temperature reading if it reports such a thing. My APC Smart-UPS 1000VA suffered a failed battery which wasn’t totally unexpected given I had last replaced it about two years ago.

While I was awaiting the replacement RBC6 battery, I had a poke about with apcupsd and noticed that it reckoned the internal temperature of the UPS was around 39-40 ℃ which seemed a bit high.

New battery arrived so I pulled the front panel off the UPS and tried to use that stupid plastic flap APC put on the battery to help you pull it out, surprise, it tore off like it’s done on every damn battery. Okay, so I’d have to reach in and get my hand behind the battery to push it out, which due to the nature of what a UPS generally does felt like that bit in the Flash Gordon movie when Peter Duncan sticks his arm in the tree stump and gets fatally stung. Nope, still not coming out and I could feel the battery was quite hot so I pulled the power and left it to cool down for an hour or two. With a bit of help from gravity it became apparent why the battery was being so stubborn:

Bursting Battery

The photo probably doesn’t illustrate just how distorted each side of the battery was. According to apcupsd the temperature is now around 25 ℃ with the new battery so it’s probably worth sticking some monitoring on that. Grabbing the temperature is easy if apcupsd(8) is running:

# /usr/sbin/apcaccess | grep -i temp
ITEMP    : 25.9 C Internal

It should be easy enough to wrap that in a simple Nagios test and/or graph it with Cacti.

This blog is IPv6-enabled

Sunday, December 5th, 2010

I’ve been quite keen on getting IPv6 connectivity to my hosts, although I’m currently lacking either a home internet package or hosted server where there is native connectivity. Thankfully there are enough free tunnel providers available which work fine as a workaround.

I’m using SixXS for two tunnels; one to my home network and another to the VPS provided by Linode which amongst other things hosts this blog.

SixXS operate a system of credits whereby tasks such as creating tunnels, obtaining subnets and setting up reverse DNS delegation costs varying amounts of credits and the only way to earn credits is to keep your tunnel up and running on a week-by-week basis. In the interests of promoting reliable and stable connectivity this is a Good Thing.

Now, what must have happened about a year ago was I created the two tunnels, but because SixXS deliberately don’t give you enough credits to be able to request subnets at the same time, you have to wait a week or two to earn sufficient credits to be able to do that. So I got my home network sorted first as that was more important and then likely got distracted by something shiny and forgot to sort out a subnet for the server.

Now, you don’t necessarily need a subnet when you have just the one host at the end a tunnel, you can just use the IPv6 address allocated to the local endpoint of the tunnel. However, you can’t change the reverse DNS of that address, meaning it will be stuck called something like cl-123.pop-01.us.sixxs.net. If you’re planning on running a service which can be picky about the reverse DNS, (SMTP is a good example), then it’s probably best to get a subnet so you have full control over it.

By this point, requesting a subnet isn’t a problem credits-wise as the tunnels have been up for over a year. My existing tunnel configuration stored under /etc/sysconfig/network-scripts/ifcfg-sit1 looked something like this:

1
2
3
4
5
6
7
8
9
10
DEVICE=sit1
BOOTPROTO=none
ONBOOT=yes
IPV6INIT=yes
IPV6_TUNNELNAME=SixXS
IPV6TUNNELIPV4=1.2.3.4
IPV6TUNNELIPV4LOCAL=5.6.7.8
IPV6ADDR=2001:dead:beef:7a::2/64
IPV6_MTU=1280
TYPE=sit

Once the subnet has been allocated and routed to your tunnel, it’s simply a case of picking an address from that and altering the configuration like so:

1
2
3
4
5
6
7
8
9
10
11
DEVICE=sit1
BOOTPROTO=none
ONBOOT=yes
IPV6INIT=yes
IPV6_TUNNELNAME=SixXS
IPV6TUNNELIPV4=1.2.3.4
IPV6TUNNELIPV4LOCAL=5.6.7.8
IPV6ADDR=2001:dead:beef:7a::2/64
IPV6ADDR_SECONDARIES="2001:feed:beef::1/128"
IPV6_MTU=1280
TYPE=sit

Note the IPV6ADDR_SECONDARIES addition. Now your host should be accessible through both addresses and thanks to #199862 any outbound connections initiated from your host should use the subnet address rather than the tunnel address. You can test this with ip(8) and some known IPv6-connected hosts.

# host -t aaaa www.sixxs.net
www.sixxs.net is an alias for nginx.sixxs.net.
nginx.sixxs.net has IPv6 address 2001:1af8:1:f006::6
nginx.sixxs.net has IPv6 address 2001:838:2:1::30:67
nginx.sixxs.net has IPv6 address 2001:960:800::2
# ip -6 route get 2001:1af8:1:f006::6
2001:1af8:1:f006::6 via 2001:1af8:1:f006::6 dev sit1  src 2001:feed:beef::1  metric 0 
    cache  mtu 1280 advmss 1220 hoplimit 4294967295
# host -t aaaa ipv6.google.com
ipv6.google.com is an alias for ipv6.l.google.com.
ipv6.l.google.com has IPv6 address 2001:4860:800f::63
# ip -6 route get 2001:4860:800f::63
2001:4860:800f::63 via 2001:4860:800f::63 dev sit1  src 2001:feed:beef::1  metric 0 
    cache  mtu 1280 advmss 1220 hoplimit 4294967295

Trying again for the remote endpoint address of the tunnel gets a different result:

# ip -6 route get 2001:dead:beef:7a::1
2001:dead:beef:7a::1 via :: dev sit1  src 2001:dead:beef:7a::2  metric 256  expires 21315606sec mtu 1280 advmss 1220 hoplimit 4294967295

The kernel decides that in this case using the local address of the tunnel is a better choice, which I think is due to RFC 3484 and how Linux chooses a source address. If that expiry counter ever hits zero and the behaviour changes, I’ll let you know…