Archive for the ‘OpenBSD’ Category

Further Soekris net6501 improvements for OpenBSD

Thursday, April 30th, 2015

The release of OpenBSD 5.7 brings the latest batch of improvements I made for Soekris net6501 users.

The first change adds platform detection that should in fact work on all Soekris boards, (I think this change actually went in circa 5.4 but I never posted about it). Normally this information comes via the SMBIOS extensions but Soekris boards have a much stripped-down BIOS that lacks any of this support. There are however some vendor and product strings embedded in the BIOS image (probably used for printing on the serial console at POST) so they get picked out and used. For example:

# sysctl hw.vendor hw.product
hw.vendor=Soekris Engineering

The second change adds the skgpio(4) driver which provides access to the GPIO & LEDs on the net6501. Two GPIO devices are provided; gpio0 for the 16 real GPIO pins, (JP8 in the net6501 manual); and gpio1 for the error and ready LEDs coerced into the GPIO framework as output-only pins. Here is what is displayed in dmesg at boot:

skgpio0 at isa0 port 0x680/32                                                  
gpio0 at skgpio0: 16 pins                                                      
gpio1 at skgpio0: 2 pins

You still need to configure the GPIO pins prior to raising the securelevel so add the following to /etc/rc.securelevel as a minimum:

gpioctl -q gpio1 0 set out error_led
gpioctl -q gpio1 1 set out ready_led

Now once the system is booted you can toggle the error LED with the following:

# gpioctl gpio1 error_led 2
pin error_led: state 0 -> 1
# gpioctl gpio1 error_led 2
pin error_led: state 1 -> 0

IPsec between Meraki and OpenBSD

Tuesday, August 6th, 2013

I recently acquired some (Cisco) Meraki networking kit including an MX60 security appliance (read: router, firewall, NAT, etc.).

Once it’s set up and running, I was browsing the dashboard and the site-to-site VPN configuration options. Normally with multiple Meraki devices in use, a fully-meshed VPN can be created automatically with very little configuration.

I also noticed the ability to add non-Meraki VPN peers so I added details for my OpenBSD-based gateway. You can see from the screenshot the details are basic:

meraki dashboard

On the OpenBSD side, I started with a basic configuration in /etc/ipsec.conf:

ike esp from to \
        peer \
        psk mekmitasdigoat

The Meraki dashboard allows you to register a dynamic DNS record for the MX60 which will be in the domain so you can use this to refer to the remote peer in the configuration, (especially if you have a dynamic IP address).

This configuration didn’t work, but using isakmpd(8)‘s handy -L option to write to /var/run/isakmpd.pcap all of the unencrypted IKE traffic it then becomes apparent the OpenBSD and the Meraki disagree on the phase 1 negotiation; mainly that the Meraki end wants to use Triple DES and OpenBSD prefers AES by default. As there’s no way to tweak the cryptography options on the Meraki side, we change the OpenBED side like so:

ike esp from to \
        peer \
        main auth hmac-sha1 enc 3des group modp1024 lifetime 28800 \
        quick auth hmac-sha1 enc aes-256 group none lifetime 28800 \
        psk mekmitasdigoat

The main option sets the phase 1 parameters and the quick option sets the phase 2 parameters that match the highest settings out of the handful proposed by the Meraki side.

With that done, all that remains is to ensure isakmpd(8) starts at boot and the rules in /etc/ipsec.conf are automatically loaded by adding the following to /etc/rc.conf.local:


Soekris CPU Scaling and OpenBSD 5.2

Tuesday, December 25th, 2012

OpenBSD 5.2 was released nearly two months ago and I had forgotten to upgrade my Soekris net6501, apart from my driver now being part of the kernel, this release also added support to the SpeedStep frequency scaling driver for the Atom CPU.

The simple benchmark to keep in mind is using the md5(1) command, using its built-in time trial test. Under the previous OpenBSD 5.1 release:

# md5 -ttt
MD5 time trial.  Processing 1000000 10000-byte blocks...
Digest = f0843f04c524250749d014a8152920ec
Time   = 124.227876 seconds
Speed  = 80497230.750367 bytes/second

You can see I’m getting roughly 80 MB/s. Now upgrade to OpenBSD 5.2 and repeat the test, you’ll get pretty much the same speeds, however now we have the CPU frequency scaling to play with. The scaling is very coarse, the only two speeds supported are 600 MHz and whatever maximum the CPU supports, so in my case with a net6501-70, it’s 1.6 GHz. To change the scaling simply manipulate the hw.setperf sysctl(8) variable.

By default the CPU is running at 100%, confirmed by:

# sysctl hw.setperf

Now set the CPU frequency to the slowest speed:

# sysctl hw.setperf=0
hw.setperf: 100 -> 0

Now running the test again I get the following results:

# md5 -ttt
MD5 time trial.  Processing 1000000 10000-byte blocks...
Digest = f0843f04c524250749d014a8152920ec
Time   = 123.948587 seconds
Speed  = 80678612.334645 bytes/second

Almost exactly the same speed, which doesn’t make sense given I’ve knocked 1 GHz off the clock speed! So put the CPU back to 100% and try again:

# sysctl hw.setperf=100
hw.setperf: 0 -> 100
# md5 -ttt
MD5 time trial.  Processing 1000000 10000-byte blocks...
Digest = f0843f04c524250749d014a8152920ec
Time   = 46.424699 seconds
Speed  = 215402581.285449 bytes/second

Ah-ha, about 210 MB/s this time! It transpires there’s a bug in the Soekris BIOS, despite advertising the CPU as 1.6 GHz it wasn’t programmed correctly and was only being clocked at 600 MHz, so all this time I’ve effectively had the base net6501-30 model albeit with the extra RAM. You can work around this by setting hw.setperf to 100 on each boot.

A new BIOS 1.41c has been released which fixes this issue and programs the CPU to run at its advertised maximum speed. However to upgrade to this involves my eternal battle with serial terminal software and uploading over XMODEM which is notoriously fickle, although I think I have it cracked…

I usually use a Mac OS X host with a KeySpan USB/Serial adapter to connect to the net6501 so I already have tools like cu(1) and screen(1). You’ll also need the lrzsz tools installed which if using MacPorts is as easy as:

# sudo port install lrzsz

Using cu(1), connect to the Soekris:

# sudo cu -l /dev/tty.KeySerial1 -s 19200

Power the board on, use Ctrl+P to break into the BIOS monitor and type download to start the Soekris waiting to receive over XMODEM. Now you need to type ~+sz -X /path/to/b6501_141c.bin, possibly as quickly as you can after the previous command. If that works, type flashupdate afterwards to reprogram the BIOS. You’ll get something like the following transcript:

> download
Start sending file using XMODEM/CRC protocol.
~+sz -X /path/to/b6501_141c.bin
Sending /path/to/b6501_141c.bin, 1982 blocks: Give your local XMODEM receive command now.
Bytes Sent: 253696   BPS:1746                            
Transfer complete
File downloaded succesfully, size 253696 Bytes.
> flashupdate
Updating BIOS Flash ,,,,,,,,,,,,, Done.
> reboot

Reboot and boot back into OpenBSD. Now the time trial should return a result of roughly 210 MB/s every time. Because I obviously don’t need 1.6 GHz of CPU all time, I’ve enabled the apmd(8) daemon which manipulates the hw.setperf variable based on the CPU idle time. Add the following to /etc/rc.conf.local:

apmd_flags="-C -f /dev/null"

The -f is only necessary when running i386 otherwise apmd(8) complains. Start with:

# /etc/rc.d/apmd start

Normally hw.setperf will be 0 however when you do something CPU-intensive (such as the MD5 time trial) apmd(8) will automatically adjust hw.setperf back to 100 so you still get the 210 MB/s result, but most of the time you’ll have lower power draw and less heat.

Soekris net6501 improvements for OpenBSD

Saturday, June 9th, 2012

I’ve recently had committed to the OpenBSD project a kernel driver, tcpcib(4), that adds support for the hardware watchdog and HPET found in the Soekris net6501, (although it should also work on any system based around the Intel Atom E600 “Tunnel Creek” processor).

I’ve not had any issue with my net6501 that has required a hard reset although I managed to trigger the watchdog a few times on the net4501 by pushing too much network traffic through it, so it’s handy to have it there as a safeguard. The HPET support simply provides a time counter with better precision that benefits anything on the system that can use it.

With the driver in your kernel, you should now see the following attach lines:

tcpcib0 at pci0 dev 31 function 0 "Intel E600 LPC" rev 0x00: 14318179 Hz timer, watchdog
isa0 at tcpcib0

Enabling the watchdog is as easy as setting kern.watchdog.period with sysctl(8). The watchdog supports a maximum period of 600 seconds and if it ever triggers a reboot of the host, you’ll be treated to a slightly different attach line:

tcpcib0 at pci0 dev 31 function 0 "Intel E600 LPC" rev 0x00: 14318179 Hz timer, watchdog, reboot on timeout

The HPET support “just works”, but you can confirm it’s active with the following:

# sysctl kern.timecounter 
kern.timecounter.choice=i8254(0) tcpcib0(2000) dummy(-1000000)

The driver is available in -current and so should be in 5.2 onwards and can be backported to 5.1 which is how I’m currently running it.

OpenBSD PPPoE and RFC 4638

Friday, January 20th, 2012

I upgraded my Internet connection from ADSL 2+ to FTTC a while ago. I’m with Eclipse as an ISP, but it’s basically the same product as BT Infinity, right down to the Openreach-branded modem, (a Huawei Echolife HG612 to be exact).

With this modem, you need to use a router or some software that can do RFC 2516 PPPoE so I simply kept using OpenBSD on my trusty Soekris net4501 and set up a pppoe(4) interface, job done. However what became apparent is the 133 MHz AMD Elan CPU couldn’t fully utilise the 40 Mb/s bandwidth I now had, at best I could get 16-20 Mb/s with a favourable wind. An upgrade was needed.

Given I’d had around 8 years of flawless service from the net4501, another Soekris board was the way to go. Enter the net6501 with comparatively loads more CPU grunt, RAM and interestingly Gigabit NIC chips; not necessarily for the faster speed, but because they can naturally handle a larger MTU.

The reason for this was that I had read that the Huawei modem and BT FTTC network fully supported RFC 4638, which means you can have an MTU of 1,500 bytes on your PPPoE connection which matches what you’ll have on your internal network. Traditionally a PPPoE connection only allowed 1,492 bytes on account of the overhead of 8 bytes of PPPoE headers in every Ethernet frame payload. Because of this it was almost mandatory to perform MSS clamping on traffic to prevent problems. So having an MTU of 1,500 bytes should avoid the need for any clamping trickery, but means your Ethernet interface needs to cope with an MTU of 1,508 bytes, hence the Gigabit NIC (which can accommodate an MTU of 9,000 bytes with no problems).

One small problem remained, pppoe(4) on OpenBSD 5.0 didn’t support RFC 4638. While I sat down and started to add support I noticed someone had added this to the NetBSD driver already, (which is where the OpenBSD driver originated from), so based on their changes I created a similar patch and with some necessary improvements based on feedback from OpenBSD developers it has now been committed to CVS in time for the 5.1 release.

To make use of the larger MTU is fairly obvious, simply set the MTU explicitly on both the Ethernet and PPPoE interfaces to 8 bytes higher than their default. As an example, my /etc/hostname.em0 now contains:

mtu 1508 up

And similarly my /etc/hostname.pppoe0 contains:

inet NONE mtu 1500 \
        pppoedev em0 authproto chap \
        authname 'user' authkey 'secret' up
!/sbin/route add default -ifp \$if

I also added support to tcpdump(8) to display the additional PPPoE tag used to negotiate the larger MTU, so when you bring the interface up, watch for PPP-Max-Payload tags going back and forth during the discovery phase.

With that done the remaining step is to remove any scrub max-mss rules in pf.conf(5) as with any luck they should no longer be required.

PPPoE fixes for natpmpd

Friday, December 16th, 2011

I recently started using the pppoe(4) driver on OpenBSD, and with it found a few small bugs in how natpmpd handles these sorts of dynamic interfaces.

One simple bug being it refused to start up if the interface didn’t already exist and also it considered as a valid IP address and would broadcast that to any clients on the network. This situation happened due to the way pppoe(4) interfaces are initially set up and would correct itself quickly once the PPPoE session was negotiated.

Both of these bugs should be fixed so natpmpd should now correctly deny any request until the interface gets created and negotiates a normal IP address, (normal Ethernet interface behaviour should be unchanged).

Alignment fixes for natpmpd

Sunday, October 24th, 2010

A one line fix and natpmpd should now work on OpenBSD platforms that have stricter code alignment requirements than i386 or amd64 such as armish and sparc64. Fairly amazed that was the only breakage.

Next release should hopefully have privilege separation now that the various imsg_*(3) functions will be easily available in the 4.8 release.

OpenBSD IPsec and RFC 3884

Sunday, October 3rd, 2010

As part of another OpenBSD & IPsec problem I’m investigating, I was pointed at RFC 3884 which puts forward a solution for solving problems with dynamic routing protocols and the use of IPsec in tunnel mode.

The RFC covers a few scenarios and solutions, but the main solution put forward is to replace IPsec tunnel mode with a combination of IPsec transport mode and the use of IP-in-IP tunnels. This has the benefit that it remains interoperable with IPsec tunnel mode implementations as the packet format is identical.

To show this you need to understand how IPsec packets are constructed. The RFC covers this and there also sites like this one which give you better diagrams. You only really need to pay attention to how ESP works rather than AH as AH is rarely used due to it’s shortcomings with the dreaded NAT. Once you know that all an IP-in-IP tunnel does is encapsulate an existing IP packet with another one you can see how the two implementations have the same wire format, the innermost packets just have a slightly different route through the network stack.

Anyway, after reading this I thought I’d test it to see if it really does work between a pair of OpenBSD hosts so I fired up a couple of stock 4.7 installs under VMware.

On both hosts I disabled pf to eliminate that as a source of any failures:

# pfctl -d

For reasons I’ll explain later, we’ll just deal with manually keyed IPsec so you’ll need to generate a handful of keys. On each host generate an authentication and encryption key:

# openssl rand 32 | hexdump -e '"0x" 32/1 "%02x" "\n"'
# openssl rand 16 | hexdump -e '"0x" 16/1 "%02x" "\n"'

On the first host we’ll set up the regular IPsec tunnel. Add the following to /etc/ipsec.conf using the keys we generated:

flow esp from to local peer
esp tunnel from to spi 0xdeadbeef:0xfeedbeef \
        authkey 0xebfd3e8038fd1c9eadcc32308e5f12aba813114614855e4a93e7407f3c40f827:0x7a5a5832c43395c62aa1c443cfda68f3f8fff89378fdb2df40419df01547a40a \
        enckey 0x46c670e4ad2ef30a29a0ce64738d01e5:0x811ce51cebcaed4280aae9cb2e195f93

We’ll need a dummy interface with the address for testing, the easiest way is to just use the loopback interface like so:

# ifconfig lo1 create
# ifconfig lo1 prefixlen 32

On the second host, we’ll need to set up the IP-in-IP tunnel which on an OpenBSD host uses the gif(4) interface:

# ifconfig gif0 create
# ifconfig gif0 tunnel
# ifconfig gif0 prefixlen 32

Now add the following to /etc/ipsec.conf using the same keys, but remember to swap around the various parameters:

flow esp proto ipencap from to
esp transport from to spi:0xfeedbeef:0xdeadbeef \
        authkey 0x7a5a5832c43395c62aa1c443cfda68f3f8fff89378fdb2df40419df01547a40a:0xebfd3e8038fd1c9eadcc32308e5f12aba813114614855e4a93e7407f3c40f827 \
        enckey 0x811ce51cebcaed4280aae9cb2e195f93:0x46c670e4ad2ef30a29a0ce64738d01e5

Note that we only apply transport mode to the IP-in-IP tunnel packets (proto ipencap) otherwise all other traffic will be dropped by the first host if you tried ping, SSH, etc. between them.

With that done, on each host now do:

# ipsecctl -f /etc/ipsec.conf

Everything should be working now so on the first host with the regular IPsec tunnel, try pinging the remote IP address. You’ll need to specify the source address so it matches the IPsec flow:

# ping -I
PING ( 56 data bytes
64 bytes from icmp_seq=0 ttl=255 time=0.628 ms
64 bytes from icmp_seq=1 ttl=255 time=0.670 ms

While that’s running, on the opposite host you should be able to use tcpdump to see the IPsec traffic flowing back and forth:

# tcpdump -nn -i vic0
tcpdump: listening on vic0, link-type EN10MB
23:43:43.067641 esp > spi 0xdeadbeef seq 23 len 136
23:43:43.067880 esp > spi 0xfeedbeef seq 55 len 136

You can also use tcpdump on the enc0 interface to see the packets passing through the IPsec stack:

# tcpdump -nn -i enc0
tcpdump: listening on enc0, link-type ENC
23:46:51.679546 (authentic,confidential): SPI 0xdeadbeef: > icmp: echo request (encap)
23:46:51.679628 (authentic,confidential): SPI 0xfeedbeef: > icmp: echo reply (encap)

Here you can see the actual payload of the packets, in this case ICMP.

You can repeat the test from the opposite side with the gif0 interface:

# ping
PING ( 56 data bytes
64 bytes from icmp_seq=0 ttl=255 time=1.008 ms
64 bytes from icmp_seq=1 ttl=255 time=0.549 ms

Note that you don’t need to specify a source address this time as the routing sorts this out and correctly assigns you the local address of the IP-in-IP tunnel interface, (this is in fact another perceived benefit of the solution proposed by the RFC).

So we’ve hopefully proved it works, however this has only been demonstrated for manually keyed IPsec associations. Most IPsec these days is set up using IKE to automatically exchange keys in various forms and dynamically configure the flows on the hosts.

Here lies the problem, if I set up each end to use isakmpd(8) neither end will agree on a proposal as one side will propose tunnel mode and the other will propose transport mode. What I need to be able to do is get the transport side to propose tunnel mode but actually configure transport mode flows on the local side and rely on a suitable gif(4) interface to be set up. Currently there doesn’t appear to be a way to do this with isakmpd(8).

Back to My Mac and an OpenBSD firewall

Friday, August 27th, 2010

As I recently wrote, I’ve been playing with the Back to My Mac feature of MobileMe on my Macs. Put simply it’s a VPN for your Macs, you can access one remotely from another as if they were on the same LAN either at home or work.

Assuming you’ve entered all of your MobileMe account details, it should just be a case of going to the Back to My Mac tab in the MobileMe preferences and starting it up on each Mac you own.

However, with the first Mac I tried this on I hit this problem:

Back to My Mac Warning Screenshot

The router that OS X is complaining about in this case is a Soekris net4501 running OpenBSD, (well to be pedantic, there’s actually two of them in a failover configuration). OpenBSD doesn’t support NAT-PMP or UPnP out of the box, so I had a look for some additional software I could run that might support either protocol.

I came across MiniUPnPd which claimed to support both protocols and run on OpenBSD so I grabbed the source and compiled it up to try it. At the time I was still running OpenBSD 4.6 but was planning to do the upgrade to 4.7 soon, I noted that there was reports of MiniUPnPd not working properly on 4.7.

After configuring it and starting it up, it didn’t seem to work properly. After eliminating any obvious reasons, the Macs still didn’t think that the router supported either NAT-PMP or UPnP, and my Sony Playstation 3 which supports UPnP only, claimed UPnP was unavailable when I ran a network diagnostic but this seems to be a known issue with the Playstation 3. So MiniUPnPd wasn’t looking too useful to me.

Out of curiosity I investigated how complicated the NAT-PMP and UPnP protocols are as the specifications for both should be publicly available. The first one I looked at was UPnP as at the time this seemed the more well known of the two. UPnP appears be a fairly bewildering set of standards, even though it seemed the bit I only need to care about is the Internet Gateway Device (IGD) protocol. It also depends heavily on XML which I loath the more I have to deal with it.

NAT-PMP on the other hand seemed a far simpler protocol, the IETF draft was straightforward by comparison and as it was authored by Apple themselves, it should be the better supported of the two, at least on my Apple hardware. After a day or two of coding, I have a fully standalone NAT-PMP daemon – natpmpd, which I’m making available under the BSD licence.

The dedicated page documents the gory details on how to get it set up, but suffice to say once installed and running, I now get the following on each Mac:

Back to My Mac Working Screenshot

To test, I simply disconnected my MacBook Pro from my home network and instead connected through my mobile phone via 3G. The remaining Macs on the home network are still visible in the Finder and Screen and File sharing remain accessible. On my OpenBSD router using tcpdump(8) I can see the encrypted VPN traffic flowing between the remote and local Macs.