Setting the date on a Muvi Micro Camcorder

February 27th, 2011

I recently got a little Veho Muvi Micro Camcorder after seeing my brother using one on his bike. I appear to have the Pro model which on the positive side can handle more frames per second, but on the negative side puts a bright green timestamp on your recordings and there’s no way to disable it.

So the best you can do is at least make sure the timestamp is correct; it will reset to some known epoch if the battery goes flat and it might lose accuracy over time although I can’t say I’ve paid attention to that. There’s a utility for Windows that ships with the device, but that’s no good on a Mac or Linux host so for the benefit of the lazyweb and myself because it’s fairly obscure how to do it…

Create a file called time.txt in the root of the Muvi filesystem (alongside the DCIM directory) with the following contents:

YYYY.MM.DD HH:MM:SS

Unmount the device and turn it off and on, the file should be consumed and the time should be now set.

Here’s a video shot in my 1955 VW Beetle which demonstrates the annoying timestamp, although does show the Muvi does just about cope with the electromagnetic interference from the magneto ignition and noise levels from the exhaust, although it doesn’t cope with the lack of daylight, blame the black paint for that.

Holy Exploding Batteries Batman

February 27th, 2011

If you have a UPS, it’s probably worth paying attention to the temperature reading if it reports such a thing. My APC Smart-UPS 1000VA suffered a failed battery which wasn’t totally unexpected given I had last replaced it about two years ago.

While I was awaiting the replacement RBC6 battery, I had a poke about with apcupsd and noticed that it reckoned the internal temperature of the UPS was around 39-40 ℃ which seemed a bit high.

New battery arrived so I pulled the front panel off the UPS and tried to use that stupid plastic flap APC put on the battery to help you pull it out, surprise, it tore off like it’s done on every damn battery. Okay, so I’d have to reach in and get my hand behind the battery to push it out, which due to the nature of what a UPS generally does felt like that bit in the Flash Gordon movie when Peter Duncan sticks his arm in the tree stump and gets fatally stung. Nope, still not coming out and I could feel the battery was quite hot so I pulled the power and left it to cool down for an hour or two. With a bit of help from gravity it became apparent why the battery was being so stubborn:

Bursting Battery

The photo probably doesn’t illustrate just how distorted each side of the battery was. According to apcupsd the temperature is now around 25 ℃ with the new battery so it’s probably worth sticking some monitoring on that. Grabbing the temperature is easy if apcupsd(8) is running:

# /usr/sbin/apcaccess | grep -i temp
ITEMP    : 25.9 C Internal

It should be easy enough to wrap that in a simple Nagios test and/or graph it with Cacti.

PlayStation 3 HDD Upgrade

December 30th, 2010

I recently got Gran Turismo 5 for my PS3 and it helpfully offered to install bits of itself to the hard disk to speed things up. I think it claimed about 10GB was needed and at the time I just accepted it. Then I got thinking, along with this I’ve blindly installed a fair bit of downloadable content for GTA IV and Dragon Age: Origins and so far nothing has told me the hard disk is full but I wondered how close I was.

A quick spot of archaeology found the box and apparently I had been a tight-arse and only purchased the 40GB model and the system settings indicated I had just over 3GB left so the next decent-sized DLC would’ve bitten me.

Thankfully I had recently upgraded my MacBook Pro with a whopping 750GB hard disk so I had the original 160GB gathering dust, plenty big enough. Sony make it really easy to upgrade the hard disk with clear instructions, all I needed to do first was perform a full backup to some external USB storage prior to swapping the hard disks over. I warn you now, this takes ages. The backup process claimed to back up 27GB (where’s my other 10GB gone?) and that took around an hour. Maybe I’m just expecting too much out of USB2?

Most of this is obvious stuff, but I remembered there was some brouhaha about some content such as game save data being non-transferable and I knew Dragon Age was one such game. I assume this is to prevent trading save games to unlock trophies as part of some sort of willy waving contest but this has the knock on effect of being unable to transfer the data if you replace your PS3 for some reason. While I was waiting for the backup to finish I considered what might be happening:

  • Does the process skip non-transferable data?
  • Does it back it up and then not restore it?
  • Does it back it up, restore it and then prevent you from loading it within the game?
  • Does a PS3 with a different hard disk count as a different system?

I assumed it wouldn’t be so silly to include the hard disk in the system profile but upon restoring the first thing I would do is check which save game data was restored and if it still worked or not, keeping the original 40GB hard disk safe so I could always downgrade again and reconsider.

Upon swapping the hardware, the PS3 correctly reformatted the hard disk which still had the OS X partitioning present and then I started the restore process which took even longer than the initial backup, I think it took about an hour and a half.

The process seems to have worked and restored everything, my Dragon Age game saves still seem to load with no problems. Weirdly the PS3 now reports that I’ve used 47GB on the new hard disk, whereas I had only used around 34GB on the old one accounting for the stupid capacity rounding hard disk manufacturers love to use, so either there’s quite the rounding error going on or the PS3 reserves a percentage of the disk for its own purposes.

This blog is IPv6-enabled

December 5th, 2010

I’ve been quite keen on getting IPv6 connectivity to my hosts, although I’m currently lacking either a home internet package or hosted server where there is native connectivity. Thankfully there are enough free tunnel providers available which work fine as a workaround.

I’m using SixXS for two tunnels; one to my home network and another to the VPS provided by Linode which amongst other things hosts this blog.

SixXS operate a system of credits whereby tasks such as creating tunnels, obtaining subnets and setting up reverse DNS delegation costs varying amounts of credits and the only way to earn credits is to keep your tunnel up and running on a week-by-week basis. In the interests of promoting reliable and stable connectivity this is a Good Thing.

Now, what must have happened about a year ago was I created the two tunnels, but because SixXS deliberately don’t give you enough credits to be able to request subnets at the same time, you have to wait a week or two to earn sufficient credits to be able to do that. So I got my home network sorted first as that was more important and then likely got distracted by something shiny and forgot to sort out a subnet for the server.

Now, you don’t necessarily need a subnet when you have just the one host at the end a tunnel, you can just use the IPv6 address allocated to the local endpoint of the tunnel. However, you can’t change the reverse DNS of that address, meaning it will be stuck called something like cl-123.pop-01.us.sixxs.net. If you’re planning on running a service which can be picky about the reverse DNS, (SMTP is a good example), then it’s probably best to get a subnet so you have full control over it.

By this point, requesting a subnet isn’t a problem credits-wise as the tunnels have been up for over a year. My existing tunnel configuration stored under /etc/sysconfig/network-scripts/ifcfg-sit1 looked something like this:

1
2
3
4
5
6
7
8
9
10
DEVICE=sit1
BOOTPROTO=none
ONBOOT=yes
IPV6INIT=yes
IPV6_TUNNELNAME=SixXS
IPV6TUNNELIPV4=1.2.3.4
IPV6TUNNELIPV4LOCAL=5.6.7.8
IPV6ADDR=2001:dead:beef:7a::2/64
IPV6_MTU=1280
TYPE=sit

Once the subnet has been allocated and routed to your tunnel, it’s simply a case of picking an address from that and altering the configuration like so:

1
2
3
4
5
6
7
8
9
10
11
DEVICE=sit1
BOOTPROTO=none
ONBOOT=yes
IPV6INIT=yes
IPV6_TUNNELNAME=SixXS
IPV6TUNNELIPV4=1.2.3.4
IPV6TUNNELIPV4LOCAL=5.6.7.8
IPV6ADDR=2001:dead:beef:7a::2/64
IPV6ADDR_SECONDARIES="2001:feed:beef::1/128"
IPV6_MTU=1280
TYPE=sit

Note the IPV6ADDR_SECONDARIES addition. Now your host should be accessible through both addresses and thanks to #199862 any outbound connections initiated from your host should use the subnet address rather than the tunnel address. You can test this with ip(8) and some known IPv6-connected hosts.

# host -t aaaa www.sixxs.net
www.sixxs.net is an alias for nginx.sixxs.net.
nginx.sixxs.net has IPv6 address 2001:1af8:1:f006::6
nginx.sixxs.net has IPv6 address 2001:838:2:1::30:67
nginx.sixxs.net has IPv6 address 2001:960:800::2
# ip -6 route get 2001:1af8:1:f006::6
2001:1af8:1:f006::6 via 2001:1af8:1:f006::6 dev sit1  src 2001:feed:beef::1  metric 0 
    cache  mtu 1280 advmss 1220 hoplimit 4294967295
# host -t aaaa ipv6.google.com
ipv6.google.com is an alias for ipv6.l.google.com.
ipv6.l.google.com has IPv6 address 2001:4860:800f::63
# ip -6 route get 2001:4860:800f::63
2001:4860:800f::63 via 2001:4860:800f::63 dev sit1  src 2001:feed:beef::1  metric 0 
    cache  mtu 1280 advmss 1220 hoplimit 4294967295

Trying again for the remote endpoint address of the tunnel gets a different result:

# ip -6 route get 2001:dead:beef:7a::1
2001:dead:beef:7a::1 via :: dev sit1  src 2001:dead:beef:7a::2  metric 256  expires 21315606sec mtu 1280 advmss 1220 hoplimit 4294967295

The kernel decides that in this case using the local address of the tunnel is a better choice, which I think is due to RFC 3484 and how Linux chooses a source address. If that expiry counter ever hits zero and the behaviour changes, I’ll let you know…

DivX playback on OS X

December 5th, 2010

I normally install the DivX software to get the support enabled in QuickTime Player via the bundled plugin.

Recently a couple of DivX-encoded movies have confused QuickTime Player such that I get the sound, but no picture although they seem to work fine in the standalone DivX Player. I’d rather not have to use multiple video playback apps so this rather smells of plugin fail.

The DivX Preferences pane forces System Preferences to relaunch in 32-bit mode, so I tried starting QuickTime Player in 32-bit mode to see if that might help, but no joy.

While searching to see what other users have done, I stumbled across Perian which is a LGPL-licenced QuickTime component that supports DivX amongst a bunch of other formats.

Installing Perian got the troublesome DivX videos working properly in QuickTime Player, and its System Preferences pane works in native 64-bit mode so it’s also slightly less annoying in that respect.

NAT-PMP fixes for Transmission

November 7th, 2010

I recently had a desire to play some old Amiga games in UAE and so I needed the Kickstart ROMs. Rather than dig out my old A1200 and somehow get the ROM image from that I attempted to just download them, (naughty, I know).

All I could find were BitTorrent links so I needed a BitTorrent client. I picked Transmission as it looked a decent OS X client and grabbed the Kickstart ROMs.

In the course of doing that I noticed a couple of small NAT-PMP related bugs, thanks to being more than slightly familiar with the protocol.

So once bugs #3727 and #3728 are fixed hopefully Transmission will play even better with natpmpd.

Alignment fixes for natpmpd

October 24th, 2010

A one line fix and natpmpd should now work on OpenBSD platforms that have stricter code alignment requirements than i386 or amd64 such as armish and sparc64. Fairly amazed that was the only breakage.

Next release should hopefully have privilege separation now that the various imsg_*(3) functions will be easily available in the 4.8 release.

Nokia Series 60 and a Cisco VPN

October 15th, 2010

One redeeming feature of Series 60 Nokia phones is there’s a reasonable IPsec VPN client available. It might be pre-installed on your phone already or you can check if your phone is compatible and download it here.

It can however be a bit of a pig to configure as you do very little of it on the phone and instead you have to supply a VPN policy file with all of the various settings contained within. Nokia ship a tool that allows you to create these, however it’s only available for Windows, so if you’re on a Mac or a Linux machine you’re out of luck. However, when you realise that a VPN policy file is just a renamed ZIP archive there’s no need to give up.

Nokia have documentation available specifically the VPN Client Policy Specification that explains what the various files in the ZIP archive are for. Unless you authenticate via certificates, chances are you only need the minimum two required files in the archive.

The first of these is a .pin file. It’s a really simple file that just supplies the policy details and is formatted like so:

1
2
3
4
5
6
7
8
9
10
[POLICYNAME]
Company VPN
[POLICYVERSION]
1.0
[POLICYDESCRIPTION]
Secure access to internal systems
[ISSUERNAME]
My Company
[CONTACTINFO]
vpn@domain.com

For the most part, you only really need to specify the policy name and version, the other fields can be empty as they aren’t generally visible.

The other required file is the .pol file. This one is a lot more complicated as it specifies all of the various IKE and IPsec parameters. A sample one looks like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
SECURITY_FILE_VERSION: 1
[INFO]
Company VPN
[POLICY]
sa CISCO_ASA_PSK = {
esp
encrypt_alg 3
auth_alg 2
identity_remote 0.0.0.0/0
src_specific
hard_lifetime_bytes 0
hard_lifetime_addtime 3600
hard_lifetime_usetime 3600
soft_lifetime_bytes 0
soft_lifetime_addtime 3600
soft_lifetime_usetime 3600
replay_win_len 0
}
 
remote 0.0.0.0 0.0.0.0 = { CISCO_ASA_PSK(vpn.domain.com) }
inbound = { } 
outbound = { }
 
[IKE]
ADDR: vpn.domain.com 255.255.255.255
IKE_VERSION: 1
MODE: Aggressive
REPLAY_STATUS: FALSE
USE_MODE_CFG: TRUE
IPSEC_EXPIRE: TRUE
USE_XAUTH: TRUE
USE_COMMIT: FALSE
ESP_UDP_PORT: 0
SEND_NOTIFICATION: TRUE
INITIAL_CONTACT: TRUE
USE_INTERNAL_ADDR: FALSE
DPD_HEARTBEAT: 90
NAT_KEEPALIVE: 60
REKEYING_THRESHOLD: 90
ID_TYPE: 11
FQDN: group
PRESHARED_KEYS:
FORMAT: STRING_FORMAT
KEY: 8 password
USE_NAT_PROBE: FALSE
PROPOSALS: 1
ENC_ALG: 3DES-CBC
AUTH_METHOD: PRE-SHARED
HASH_ALG: MD5
GROUP_DESCRIPTION: MODP_1024
GROUP_TYPE: DEFAULT
LIFETIME_KBYTES: 0
LIFETIME_SECONDS: 86400
PRF: NONE

When creating this file it’s handy to know what the other side is expecting otherwise a lot of trial and error is involved.

The above file works for me to connect to a Cisco ASA firewall, some key points:

  • Lines 7 & 8 specify the IPsec crypto algorithms, in this case 3DES-MD5. To use for example AES-SHA1 instead you would change the values on those lines to 12 and 3 respectively.
  • Line 20 specifies the address of your VPN server, this can be either a fully-qualified domain name or an IP address.
  • Line 25 repeats the address of your VPN server, except this time you also need to specify the netmask, which in most cases is 255.255.255.255.
  • Line 31 enables XAUTH which is required in common Cisco setups where you authenticate with your own personal username and password which is quite often linked to Active Directory, etc.
  • Lines 40 & 41 specify the first authentication step which in my setup is a shared group and password, so ID type 11 is used to specify that FQDN is just bytes rather than an IP address or fully-qualified domain, etc. then the group name itself is specified on the following line.
  • Line 44 specifies the password, note that you first specify the length of the password followed by the actual password string itself.
  • Lines 47 & 49 specify the IKE crypto algorithms. In contrast to the IPsec values these are actually spelled out instead of using cryptic values.

It’s important that both of these files are saved in DOS format with the correct newlines. You can use for example unix2dos(1) to convert them:

# unix2dos policy.p*

Then all that is left is to create the zip archive:

# zip policy.vpn policy.p*

It’s important that all three files share a common basename, in this example we used “policy” but it can be anything.

Now you need to upload the .vpn file to the phone. This can be done in any number of ways, such as USB or Bluetooth. You then need to run the file on the phone which forces it to be installed as a VPN policy.

Once that’s done you need to create a VPN access point which is accessed on my Nokia E71 under Tools > Settings > Connection > VPN > VPN access points. There you create a new access point, you need to give it a name, pick the policy that you just installed and an existing access point for it to piggyback on top of such as your 3G/GPRS connection or WiFi.

With the VPN access point defined you now use it like a regular access point. One thing I haven’t managed to get working yet is split tunnelling so that while I’m using the VPN I can access regular websites, instead I have to reconnect with a non-VPN access point but this is a minor niggle.

OpenBSD IPsec and RFC 3884

October 3rd, 2010

As part of another OpenBSD & IPsec problem I’m investigating, I was pointed at RFC 3884 which puts forward a solution for solving problems with dynamic routing protocols and the use of IPsec in tunnel mode.

The RFC covers a few scenarios and solutions, but the main solution put forward is to replace IPsec tunnel mode with a combination of IPsec transport mode and the use of IP-in-IP tunnels. This has the benefit that it remains interoperable with IPsec tunnel mode implementations as the packet format is identical.

To show this you need to understand how IPsec packets are constructed. The RFC covers this and there also sites like this one which give you better diagrams. You only really need to pay attention to how ESP works rather than AH as AH is rarely used due to it’s shortcomings with the dreaded NAT. Once you know that all an IP-in-IP tunnel does is encapsulate an existing IP packet with another one you can see how the two implementations have the same wire format, the innermost packets just have a slightly different route through the network stack.

Anyway, after reading this I thought I’d test it to see if it really does work between a pair of OpenBSD hosts so I fired up a couple of stock 4.7 installs under VMware.

On both hosts I disabled pf to eliminate that as a source of any failures:

# pfctl -d

For reasons I’ll explain later, we’ll just deal with manually keyed IPsec so you’ll need to generate a handful of keys. On each host generate an authentication and encryption key:

# openssl rand 32 | hexdump -e '"0x" 32/1 "%02x" "\n"'
0x7a5a5832c43395c62aa1c443cfda68f3f8fff89378fdb2df40419df01547a40a
# openssl rand 16 | hexdump -e '"0x" 16/1 "%02x" "\n"'
0x811ce51cebcaed4280aae9cb2e195f93

On the first host we’ll set up the regular IPsec tunnel. Add the following to /etc/ipsec.conf using the keys we generated:

flow esp from 10.0.0.1/32 to 10.0.0.2/32 local 172.16.252.128 peer 172.16.252.130
esp tunnel from 172.16.252.128 to 172.16.252.130 spi 0xdeadbeef:0xfeedbeef \
        authkey 0xebfd3e8038fd1c9eadcc32308e5f12aba813114614855e4a93e7407f3c40f827:0x7a5a5832c43395c62aa1c443cfda68f3f8fff89378fdb2df40419df01547a40a \
        enckey 0x46c670e4ad2ef30a29a0ce64738d01e5:0x811ce51cebcaed4280aae9cb2e195f93

We’ll need a dummy interface with the 10.0.0.1 address for testing, the easiest way is to just use the loopback interface like so:

# ifconfig lo1 create
# ifconfig lo1 10.0.0.1 prefixlen 32

On the second host, we’ll need to set up the IP-in-IP tunnel which on an OpenBSD host uses the gif(4) interface:

# ifconfig gif0 create
# ifconfig gif0 tunnel 172.16.252.130 172.16.252.128
# ifconfig gif0 10.0.0.2 10.0.0.1 prefixlen 32

Now add the following to /etc/ipsec.conf using the same keys, but remember to swap around the various parameters:

flow esp proto ipencap from 172.16.252.130 to 172.16.252.128
esp transport from 172.16.252.130 to 172.16.252.128 spi:0xfeedbeef:0xdeadbeef \
        authkey 0x7a5a5832c43395c62aa1c443cfda68f3f8fff89378fdb2df40419df01547a40a:0xebfd3e8038fd1c9eadcc32308e5f12aba813114614855e4a93e7407f3c40f827 \
        enckey 0x811ce51cebcaed4280aae9cb2e195f93:0x46c670e4ad2ef30a29a0ce64738d01e5

Note that we only apply transport mode to the IP-in-IP tunnel packets (proto ipencap) otherwise all other traffic will be dropped by the first host if you tried ping, SSH, etc. between them.

With that done, on each host now do:

# ipsecctl -f /etc/ipsec.conf

Everything should be working now so on the first host with the regular IPsec tunnel, try pinging the remote IP address. You’ll need to specify the source address so it matches the IPsec flow:

# ping -I 10.0.0.1 10.0.0.2
PING 10.0.0.2 (10.0.0.2): 56 data bytes
64 bytes from 10.0.0.2: icmp_seq=0 ttl=255 time=0.628 ms
64 bytes from 10.0.0.2: icmp_seq=1 ttl=255 time=0.670 ms
...

While that’s running, on the opposite host you should be able to use tcpdump to see the IPsec traffic flowing back and forth:

# tcpdump -nn -i vic0
tcpdump: listening on vic0, link-type EN10MB
23:43:43.067641 esp 172.16.252.128 > 172.16.252.130 spi 0xdeadbeef seq 23 len 136
23:43:43.067880 esp 172.16.252.130 > 172.16.252.128 spi 0xfeedbeef seq 55 len 136
...

You can also use tcpdump on the enc0 interface to see the packets passing through the IPsec stack:

# tcpdump -nn -i enc0
tcpdump: listening on enc0, link-type ENC
23:46:51.679546 (authentic,confidential): SPI 0xdeadbeef: 10.0.0.1 > 10.0.0.2: icmp: echo request (encap)
23:46:51.679628 (authentic,confidential): SPI 0xfeedbeef: 10.0.0.2 > 10.0.0.1: icmp: echo reply (encap)
...

Here you can see the actual payload of the packets, in this case ICMP.

You can repeat the test from the opposite side with the gif0 interface:

# ping 10.0.0.1
PING 10.0.0.1 (10.0.0.1): 56 data bytes
64 bytes from 10.0.0.1: icmp_seq=0 ttl=255 time=1.008 ms
64 bytes from 10.0.0.1: icmp_seq=1 ttl=255 time=0.549 ms
...

Note that you don’t need to specify a source address this time as the routing sorts this out and correctly assigns you the local address of the IP-in-IP tunnel interface, (this is in fact another perceived benefit of the solution proposed by the RFC).

So we’ve hopefully proved it works, however this has only been demonstrated for manually keyed IPsec associations. Most IPsec these days is set up using IKE to automatically exchange keys in various forms and dynamically configure the flows on the hosts.

Here lies the problem, if I set up each end to use isakmpd(8) neither end will agree on a proposal as one side will propose tunnel mode and the other will propose transport mode. What I need to be able to do is get the transport side to propose tunnel mode but actually configure transport mode flows on the local side and rely on a suitable gif(4) interface to be set up. Currently there doesn’t appear to be a way to do this with isakmpd(8).

Issues with iptables stateful filtering

September 29th, 2010

I hit a weird issue today, I have Apache configured as a reverse proxy using mod_proxy_balancer which is a welcome addition in Apache 2.2.x. This is forwarding selected requests to more Apache instances running mod_perl, although this could be any sort of application layer, pretty standard stuff.

With ProxyStatus enabled and pushing a reasonable amount of traffic through the proxy, I started to notice that the application layer Apache instances would consistently get marked in error state every so often, removing them from the pool of available servers until their cooldown period expired and the proxy enabled them again.

Investigating the logs showed up this error:

[Tue Sep 28 00:24:39 2010] [error] (113)No route to host: proxy: HTTP: attempt to connect to 192.0.2.1:80 (192.0.2.1) failed
[Tue Sep 28 00:24:39 2010] [error] ap_proxy_connect_backend disabling worker for (192.0.2.1)

A spot of the ol’ google-fu turned up descriptions of similar problems, some not even related to Apache. The problem looked related to iptables.

All the servers are running CentOS 5 and anyone who runs this is probably aware of the stock Red Hat iptables ruleset. With HTTP access enabled, it looks something similar to this:

1
2
3
4
5
6
7
8
9
10
11
12
13
-A INPUT -j RH-Firewall-1-INPUT
-A FORWARD -j RH-Firewall-1-INPUT
-A RH-Firewall-1-INPUT -i lo -j ACCEPT
-A RH-Firewall-1-INPUT -p icmp --icmp-type any -j ACCEPT
-A RH-Firewall-1-INPUT -p 50 -j ACCEPT
-A RH-Firewall-1-INPUT -p 51 -j ACCEPT
-A RH-Firewall-1-INPUT -p udp --dport 5353 -d 224.0.0.251 -j ACCEPT
-A RH-Firewall-1-INPUT -p udp -m udp --dport 631 -j ACCEPT
-A RH-Firewall-1-INPUT -p tcp -m tcp --dport 631 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 22 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 80 -j ACCEPT
-A RH-Firewall-1-INPUT -j REJECT --reject-with icmp-host-prohibited

Almost all of it is boilerplate apart from line 12, which I added and is identical to the line above it granting access to SSH. Analysing the live hit counts against each of these rules showed a large number hitting that last catch-all rule on line 13 and indeed, this is what is causing the Apache errors.

Analysing the traffic with tcpdump/wireshark showed that the frontend Apache server is only getting as far as sending the initial SYN packet and it’s failing to match either the dedicated rule on line 12 for HTTP traffic, or even the rule on line 10 to match any related or previously established traffic, although I wouldn’t really expect it to match that.

Adding a rule before the last one to match and log any HTTP packets that are considered to be in the state INVALID showed that indeed for some strange reason, iptables is deciding that an initial SYN is somehow invalid.

More information about why it might be invalid can be coaxed from the kernel by issuing the following:

# echo 255 > /proc/sys/net/ipv4/netfilter/ip_conntrack_log_invalid

Although all this gave me was some extra text saying “invalid state” and the same output you get from the standard logging target:

ip_ct_tcp: invalid state IN= OUT= SRC=192.0.2.2 DST=192.0.2.1 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=21158 DF PROTO=TCP SPT=57351 DPT=80 SEQ=1402246598 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405B40402080A291DC7600000000001030307)

From searching the netfilter bugzilla for any matching bug reports I found a knob that relaxes the connection tracking, enabled with the following:

# echo 1 > /proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_be_liberal

This didn’t fix things, plus it’s recommended to only use this in extreme cases with a broken router or firewall. As all of these machines are on the same network segment with just a switch connecting them there shouldn’t be anything mangling the packets.

With those avenues exhausted the suggested workaround of adding rules similar to the following:

1
-A RH-Firewall-1-INPUT -m tcp -p tcp --dport 80 --syn -j REJECT --reject-with tcp-reset

before your last rule doesn’t really work as the proxy still drops the servers from the pool as before, but just logs a different reason instead:

[Tue Sep 28 13:59:23 2010] [error] (111)Connection refused: proxy: HTTP: attempt to connect to 192.0.2.1:80 (192.0.2.1) failed
[Tue Sep 28 13:59:23 2010] [error] ap_proxy_connect_backend disabling worker for (192.0.2.1)

This isn’t really acceptable to me anyway as it requires your proxy to retry in response to a server politely telling it to go away and it’s going to be a performance hit. The whole point of using mod_proxy_balancer for me was so it could legitimately remove servers that are dead or administratively stopped, how is it supposed to tell the difference between that and a dodgy firewall?

The only solution that worked and was deemed acceptable was to simply remove the state requirement on matching the HTTP traffic, like so:

-A RH-Firewall-1-INPUT -m tcp -p tcp --dport 80 -j ACCEPT

This will match both new and invalid packets, however it’s left me with a pretty low opinion of iptables now. Give me pf any day.