Posts Tagged ‘Cisco’

IPsec between Meraki and OpenBSD

Tuesday, August 6th, 2013

I recently acquired some (Cisco) Meraki networking kit including an MX60 security appliance (read: router, firewall, NAT, etc.).

Once it’s set up and running, I was browsing the dashboard and the site-to-site VPN configuration options. Normally with multiple Meraki devices in use, a fully-meshed VPN can be created automatically with very little configuration.

I also noticed the ability to add non-Meraki VPN peers so I added details for my OpenBSD-based gateway. You can see from the screenshot the details are basic:

meraki dashboard

On the OpenBSD side, I started with a basic configuration in /etc/ipsec.conf:

ike esp from to \
        peer \
        psk mekmitasdigoat

The Meraki dashboard allows you to register a dynamic DNS record for the MX60 which will be in the domain so you can use this to refer to the remote peer in the configuration, (especially if you have a dynamic IP address).

This configuration didn’t work, but using isakmpd(8)‘s handy -L option to write to /var/run/isakmpd.pcap all of the unencrypted IKE traffic it then becomes apparent the OpenBSD and the Meraki disagree on the phase 1 negotiation; mainly that the Meraki end wants to use Triple DES and OpenBSD prefers AES by default. As there’s no way to tweak the cryptography options on the Meraki side, we change the OpenBED side like so:

ike esp from to \
        peer \
        main auth hmac-sha1 enc 3des group modp1024 lifetime 28800 \
        quick auth hmac-sha1 enc aes-256 group none lifetime 28800 \
        psk mekmitasdigoat

The main option sets the phase 1 parameters and the quick option sets the phase 2 parameters that match the highest settings out of the handful proposed by the Meraki side.

With that done, all that remains is to ensure isakmpd(8) starts at boot and the rules in /etc/ipsec.conf are automatically loaded by adding the following to /etc/rc.conf.local:


Elasticsearch clusters plugged into Cisco switches

Friday, May 31st, 2013

I was building up an Elasticsearch cluster as a storage backend for Logstash. Using the default Zen discovery method, I found that some of the nodes in the cluster could not automatically find each other.

Zen uses multicast to locate other nodes and by using tcpdump I could see that some nodes weren’t receiving the multicast traffic. This lead me straight to the network layer and I found the culprit; a Cisco Nexus 5000 switch.

Normally a switch often treats multicast traffic much like broadcast traffic, it floods the packet to every port on the same VLAN. However Cisco switches try to be clever and learn which ports are interested in receiving the traffic by listening for IGMP packets, (referred to as snooping), but IGMP is only sent if there’s a multicast router on the network, (note also I’m not trying to route multicast, all nodes are on the same VLAN).

The solution was to enable a feature on the Nexus known as an “IGMP querier”. What this does is mimic enough of a multicast router that nodes report which multicast groups they’re interested in receiving traffic for and the switch can then learn which ports to forward multicast traffic on.

On the Nexus 5000, I needed to add the following configuration:

vlan configuration 1234
  ip igmp snooping querier

(If you have a Catalyst switch running IOS, the configuration should be very similar)

The VLAN should match whatever VLAN the nodes are attached to, and you can basically make up the IP address used here, the switch sends IGMP packets with it as the source, but it’s never used as the destination for packets, nodes use a specific multicast group instead.

As soon as I added this configuration my Elasticsearch cluster sprung into life.

Using SNMP to upgrade IOS on Cisco devices

Friday, April 29th, 2011

Following on from my previous post regarding using SNMP to initiate TFTP backups, it’s also possible to use SNMP to remotely upgrade IOS on a Cisco device.

I’ll reuse and extend the setup and configuration from the previous article so apply that first. The IOS device then needs the following additions:

router(config)#snmp-server view myview ciscoFlashOps included
router(config)#snmp-server view myview lts.9 included
router(config)#snmp-server system-shutdown

The first command grants SNMP write access to allow us to manipulate the flash storage in the device. The remaining commands allow us to trigger a reload/reboot of the device remotely via SNMP.

On our “management station” we’ll need a few more MIB files installing under ~/mibs or wherever you put them:


Update the $MIBS environment variables to include them, you’ll need something like the following:


Now, my devices don’t have enough flash storage to hold two IOS images so I normally have to delete the current version (which is loaded into memory, so it doesn’t affect the operation of the device) to free up space and then upload the new version, reload, and pray it works.

To find the filename of the current IOS image we can use the SNMP equivalent of dir flash:/:

# snmptable -v 3 -l authPriv -a MD5 -A authsecret -x DES -X privsecret -u myuser -Cb -Ci -Cw 80 ciscoFlashFileTable
SNMP table: CISCO-FLASH-MIB::ciscoFlashFileTable
 index           Size     Checksum Status                                Name
 1.1.1      576 bytes        "0x0"  valid                                   /
 1.1.2 16459360 bytes "0xDEADBEEF"  valid c870-advsecurityk9-mz.124-15.T9.bin
SNMP table CISCO-FLASH-MIB::ciscoFlashFileTable, part 2
 index      Type Date
 1.1.1 directory    ?
 1.1.2   unknown    ?

The index is a composite of the flash device, partition and file, so 1.1.2 is the second file on the first partition of the first flash device. Unless you have multiple flash devices and/or partitions you can largely ignore these details, take a look at ciscoFlashDeviceTable & ciscoFlashPartitionTable if you want more information.

With the filename, we now need to construct a request to delete it. This sort of operation along with tasks such as erasing the flash, etc. uses the ciscoFlashMiscOpTable SNMP table. Using the same technique I demonstrated in the previous article insert a new row and activate it:

# snmpset -v 3 -l authPriv -a MD5 -A authsecret -x DES -X privsecret -u myuser \
ciscoFlashMiscOpCommand.23 i delete \
ciscoFlashMiscOpDestinationName.23 s c870-advsecurityk9-mz.124-15.T9.bin \
ciscoFlashMiscOpEntryStatus.23 i createAndGo
CISCO-FLASH-MIB::ciscoFlashMiscOpCommand.23 = INTEGER: delete(3)
CISCO-FLASH-MIB::ciscoFlashMiscOpDestinationName.23 = STRING: c870-advsecurityk9-mz.124-15.T9.bin
CISCO-FLASH-MIB::ciscoFlashMiscOpEntryStatus.23 = INTEGER: createAndGo(4)

Note that I’m using createAndGo instead of active for the row status because it’s part of the same initial set request. Now check the table to see if the operation was successful:

# snmptable -v 3 -l authPriv -a MD5 -A authsecret -x DES -X privsecret -u myuser -Cb -Ci -Cw 80 ciscoFlashMiscOpTableSNMP table: CISCO-FLASH-MIB::ciscoFlashMiscOpTable
 index Command                     DestinationName                 Status
    23  delete c870-advsecurityk9-mz.124-15.T9.bin miscOpOperationSuccess
SNMP table CISCO-FLASH-MIB::ciscoFlashMiscOpTable, part 2
 index NotifyOnCompletion         Time EntryStatus
    23              false 0:0:00:56.24      active

It worked so checking the flash storage should confirm the file is now absent. Now we can upload the new IOS image, you’ll need to have it copied to /tftpboot on your TFTP server. This uses another SNMP table but the procedure is exactly the same, insert a row describing the operation we want to perform:

# snmpset -v 3 -l authPriv -a MD5 -A authsecret -x DES -X privsecret -u myuser \
ciscoFlashCopyCommand.23 i copyToFlashWithoutErase \
ciscoFlashCopyProtocol.23 i tftp \
ciscoFlashCopyServerAddress.23 a \
ciscoFlashCopySourceName.23 s c870-advsecurityk9-mz.124-24.T4.bin \
ciscoFlashCopyEntryStatus.23 i createAndGo
CISCO-FLASH-MIB::ciscoFlashCopyCommand.23 = INTEGER: copyToFlashWithoutErase(2)
CISCO-FLASH-MIB::ciscoFlashCopyProtocol.23 = INTEGER: tftp(1)
CISCO-FLASH-MIB::ciscoFlashCopyServerAddress.23 = IpAddress:
CISCO-FLASH-MIB::ciscoFlashCopySourceName.23 = STRING: c870-advsecurityk9-mz.124-24.T4.bin
CISCO-FLASH-MIB::ciscoFlashCopyEntryStatus.23 = INTEGER: createAndGo(4)

Chances are this operation will take a minute or two to complete, keep checking ciscoFlashCopyTable until it completes:

# snmptable -v 3 -l authPriv -a MD5 -A authsecret -x DES -X privsecret -u myuser -Cb -Ci -Cw 80 ciscoFlashCopyTable
SNMP table: CISCO-FLASH-MIB::ciscoFlashCopyTable
 index                 Command Protocol ServerAddress
    23 copyToFlashWithoutErase     tftp
SNMP table CISCO-FLASH-MIB::ciscoFlashCopyTable, part 2
 index                          SourceName DestinationName RemoteUserName
    23 c870-advsecurityk9-mz.124-24.T4.bin                               
SNMP table CISCO-FLASH-MIB::ciscoFlashCopyTable, part 3
 index               Status NotifyOnCompletion         Time EntryStatus Verify
    23 copyOperationSuccess              false 0:0:05:03.71      active  false
SNMP table CISCO-FLASH-MIB::ciscoFlashCopyTable, part 4
 index ServerAddrType ServerAddrRev1 RemotePassword
    23           ipv4    ""              ?

With the new image uploaded it’s time for the coup de grĂ¢ce, reloading the device:

# snmpset -v 3 -l authPriv -a MD5 -A authsecret -x DES -X privsecret -u myuser \
tsMsgSend.0 i reload
OLD-CISCO-TS-MIB::tsMsgSend.0 = INTEGER: reload(2)

Your device should now reboot and after a short while come back up running the updated IOS image. You can confirm the source of the reboot request with the following:

router#show version | i reason
Last reload reason: snmp shutdown request

Powerful stuff. Needless to say, don’t make this accessible to the world with an SNMPv2 community of “private”.

Using SNMP to trigger Cisco TFTP backups

Sunday, April 24th, 2011

I recently discovered you can use SNMP to trigger a Cisco device to write its configuration to a remote TFTP server. Apparently this feature has existed in Cisco devices in one form or another for over a decade, but somehow I’d missed it completely. This seems a far better alternative to getting in a muddle with Expect-style scripts to telnet or SSH into the device. One or more Access Control Lists can also be utilised to restrict access.

First of all, let’s configure the IOS device which will need to be running at least IOS 12.0. The first thing to do is create an Access Control List which matches the IP address of the “management station” which will be issuing the SNMP commands and for simplicities sake is also the TFTP server:

router(config)#access-list 20 permit host
router(config)#access-list 20 deny any

Next we define an SNMP view that restricts how much of the tree is writable, allowing write access to everything is both unnecessary and inadvisable:

router(config)#snmp-server view myview ccCopyTable included

Now we create an SNMP group that ties the ACL and view together:

router(config)#snmp-server group mygroup v3 priv write myview access 20

Note that I’ve specified SNMPv3 along with the authPriv security level, not all devices support that so adjust accordingly. I’ve also not specified a read view so it will use the default v1default view which can read pretty much everything. Next create a new user as a member of this group:

router(config)#snmp-server user myuser mygroup v3 auth md5 authsecret priv des privsecret

Finally, restrict which TFTP servers can be written to, triggered by SNMP. This stops someone from making your device write files to an arbitrary TFTP server somewhere:

router(config)#snmp-server tftp-server-list 20

That should be all of the configuration needed for the Cisco device.

Now on our “management station” which can be anything provided it has a recent version of Net-SNMP installed, (CentOS 5.x works fine), first make sure TFTP is installed and running. Normally the TFTP server cannot create new files so create an empty file ready:

# touch /tftpboot/router.conf

Now install the various MIB files we’ll need. You don’t need these but for the sake of readability it’s easier to use the symbolic names rather than raw OID strings. You’ll need to download the following MIB files:


Put them in a directory such as ~/mibs and then configure Net-SNMP with the following two environment variables:

# export MIBDIRS=+${HOME}/mibs

You should be able to test that everything is configured correctly:

# snmptable -v 3 -l authPriv -a MD5 -A authsecret -x DES -X privsecret -u myuser ccCopyTable
CISCO-CONFIG-COPY-MIB::ccCopyTable: No entries

Now we need to create a row in this table describing the copy operation we’d like to perform. Without getting too heavy into how SNMP tables work, we basically need to use a unique index number to refer to the row. As we’ve already shown, the table is empty so pick any random number, umm, 23! Create a new row with the following column values:

# snmpset -v 3 -l authPriv -a MD5 -A authsecret -x DES -X privsecret -u myuser \
> ccCopyProtocol.23 i tftp \
> ccCopySourceFileType.23 i runningConfig \
> ccCopyDestFileType.23 i networkFile \
> ccCopyServerAddress.23 a \
> ccCopyFileName.23 s router.conf
CISCO-CONFIG-COPY-MIB::ccCopyProtocol.23 = INTEGER: tftp(1)
CISCO-CONFIG-COPY-MIB::ccCopySourceFileType.23 = INTEGER: runningConfig(4)
CISCO-CONFIG-COPY-MIB::ccCopyDestFileType.23 = INTEGER: networkFile(1)
CISCO-CONFIG-COPY-MIB::ccCopyServerAddress.23 = IpAddress:
CISCO-CONFIG-COPY-MIB::ccCopyFileName.23 = STRING: router.conf

Now if you re-run the snmptable command (with some extra flagsoup to help readability) you’ll see there’s now a row:

# snmptable -v 3 -l authPriv -a MD5 -A authsecret -x DES -X privsecret -u myuser -Cb -Ci -Cw 80 ccCopyTable
 index Protocol SourceFileType DestFileType ServerAddress    FileName UserName
    23     tftp  runningConfig  networkFile router.conf        ?
SNMP table CISCO-CONFIG-COPY-MIB::ccCopyTable, part 2
 index UserPassword NotificationOnCompletion State TimeStarted TimeCompleted
    23            ?                    false     ?           ?             ?
SNMP table CISCO-CONFIG-COPY-MIB::ccCopyTable, part 3
 index FailCause EntryRowStatus ServerAddressType ServerAddressRev1
    23         ?              ?              ipv4       ""

This won’t initiate the operation until we set one more column to activate the table row:

# snmpset -v 3 -l authPriv -a MD5 -A authsecret -x DES -X privsecret -u myuser \
> ccCopyEntryRowStatus.23 i active
CISCO-CONFIG-COPY-MIB::ccCopyEntryRowStatus.23 = INTEGER: active(1)

Now check the table again:

# snmptable -v 3 -l authPriv -a MD5 -A authsecret -x DES -X authsecret -u myuser -Cb -Ci -Cw 80 ccCopyTable
 index Protocol SourceFileType DestFileType ServerAddress    FileName UserName
    23     tftp  runningConfig  networkFile router.conf        ?
SNMP table CISCO-CONFIG-COPY-MIB::ccCopyTable, part 2
 index UserPassword NotificationOnCompletion      State  TimeStarted
    23            ?                    false successful 5:0:20:49.92
SNMP table CISCO-CONFIG-COPY-MIB::ccCopyTable, part 3
 index TimeCompleted FailCause EntryRowStatus ServerAddressType
    23  5:0:20:51.39         ?         active              ipv4
SNMP table CISCO-CONFIG-COPY-MIB::ccCopyTable, part 4
 index ServerAddressRev1
    23       ""

You can see the state column now reads successful so check the empty file you created under /tftpboot, it should now have the contents of the current configuration for your device.

The row in the table will be automatically removed after a few minutes but you can also explicitly remove it:

# snmpset -v 3 -l authPriv -a MD5 -A authsecret -x DES -X privsecret -u myuser \
> ccCopyEntryRowStatus.23 i destroy
CISCO-CONFIG-COPY-MIB::ccCopyEntryRowStatus.23 = INTEGER: destroy(6)
# snmptable -v 3 -l authPriv -a MD5 -A authsecret -x DES -X privsecret -u myuser ccCopyTable
CISCO-CONFIG-COPY-MIB::ccCopyTable: No entries

There’s still a fair bit of functionality that I’ve skipped over, you might be able to use something other than TFTP or generate an SNMP trap on completion. Experiment with changing the Access Control Lists and stopping the TFTP server to see how the table output changes.

Nokia Series 60 and a Cisco VPN

Friday, October 15th, 2010

One redeeming feature of Series 60 Nokia phones is there’s a reasonable IPsec VPN client available. It might be pre-installed on your phone already or you can check if your phone is compatible and download it here.

It can however be a bit of a pig to configure as you do very little of it on the phone and instead you have to supply a VPN policy file with all of the various settings contained within. Nokia ship a tool that allows you to create these, however it’s only available for Windows, so if you’re on a Mac or a Linux machine you’re out of luck. However, when you realise that a VPN policy file is just a renamed ZIP archive there’s no need to give up.

Nokia have documentation available specifically the VPN Client Policy Specification that explains what the various files in the ZIP archive are for. Unless you authenticate via certificates, chances are you only need the minimum two required files in the archive.

The first of these is a .pin file. It’s a really simple file that just supplies the policy details and is formatted like so:

Company VPN
Secure access to internal systems
My Company

For the most part, you only really need to specify the policy name and version, the other fields can be empty as they aren’t generally visible.

The other required file is the .pol file. This one is a lot more complicated as it specifies all of the various IKE and IPsec parameters. A sample one looks like this:

Company VPN
encrypt_alg 3
auth_alg 2
hard_lifetime_bytes 0
hard_lifetime_addtime 3600
hard_lifetime_usetime 3600
soft_lifetime_bytes 0
soft_lifetime_addtime 3600
soft_lifetime_usetime 3600
replay_win_len 0
remote = { CISCO_ASA_PSK( }
inbound = { } 
outbound = { }
MODE: Aggressive
FQDN: group
KEY: 8 password

When creating this file it’s handy to know what the other side is expecting otherwise a lot of trial and error is involved.

The above file works for me to connect to a Cisco ASA firewall, some key points:

  • Lines 7 & 8 specify the IPsec crypto algorithms, in this case 3DES-MD5. To use for example AES-SHA1 instead you would change the values on those lines to 12 and 3 respectively.
  • Line 20 specifies the address of your VPN server, this can be either a fully-qualified domain name or an IP address.
  • Line 25 repeats the address of your VPN server, except this time you also need to specify the netmask, which in most cases is
  • Line 31 enables XAUTH which is required in common Cisco setups where you authenticate with your own personal username and password which is quite often linked to Active Directory, etc.
  • Lines 40 & 41 specify the first authentication step which in my setup is a shared group and password, so ID type 11 is used to specify that FQDN is just bytes rather than an IP address or fully-qualified domain, etc. then the group name itself is specified on the following line.
  • Line 44 specifies the password, note that you first specify the length of the password followed by the actual password string itself.
  • Lines 47 & 49 specify the IKE crypto algorithms. In contrast to the IPsec values these are actually spelled out instead of using cryptic values.

It’s important that both of these files are saved in DOS format with the correct newlines. You can use for example unix2dos(1) to convert them:

# unix2dos policy.p*

Then all that is left is to create the zip archive:

# zip policy.vpn policy.p*

It’s important that all three files share a common basename, in this example we used “policy” but it can be anything.

Now you need to upload the .vpn file to the phone. This can be done in any number of ways, such as USB or Bluetooth. You then need to run the file on the phone which forces it to be installed as a VPN policy.

Once that’s done you need to create a VPN access point which is accessed on my Nokia E71 under Tools > Settings > Connection > VPN > VPN access points. There you create a new access point, you need to give it a name, pick the policy that you just installed and an existing access point for it to piggyback on top of such as your 3G/GPRS connection or WiFi.

With the VPN access point defined you now use it like a regular access point. One thing I haven’t managed to get working yet is split tunnelling so that while I’m using the VPN I can access regular websites, instead I have to reconnect with a non-VPN access point but this is a minor niggle.

Snow Leopard and (uninstalling) Cisco VPN

Saturday, August 14th, 2010

I mentioned reinstalling Shimo when I upgraded to Snow Leopard and that works really nicely for controlling my Cisco VPN connection but for that to work, you still need to install the official Cisco VPN client.

I also use VMware Fusion and find both this and the Cisco VPN client have a propensity to step on each others toes in the networking department, my hunch is it’s most likely because both create and monitor virtual network interfaces. The usual symptoms are that my virtual machines lose network connectivity or cannot access the VPN, or that I can’t bring the VPN connection up in the first place. Sometimes I can restore order to the VPN by restarting it with the following in the Terminal:

$ sudo /System/Library/StartupItems/CiscoVPN/CiscoVPN restart

Anyway, while I was tethering my Mac to my mobile phone I was fiddling with the Network preferences and noticed Snow Leopard gained native support for CIsco VPN connections:

Cisco VPN Interface Screenshot

This was probably in the release notes which I didn’t read, (well-known male trait), but anyway adding this new interface and configuring it with the right settings got it working first, second, third time once I remembered the correct password. It even feels quicker.

That left me with little need for Shimo or the Cisco VPN client. Uninstalling Shimo was easy, I removed it from my Login Items under Accounts preferences and then dragged it from the Applications folder to the trash.

Uninstalling the Cisco VPN client was harder as that came from an .mpkg file but then I remembered something about these leaving package receipts somewhere. Sure enough, under /Library/Receipts was a handful of vpnclient-*.pkg files. A bit of a google suggested each contains a .bom file that you can inspect with lsbom(8), I found the following worked in the Terminal:

$ lsbom /Library/Receipts/vpnclient-api.pkg/Contents/Resources/
.	40755	502/0
./Library	40755	502/0
./Library/Frameworks	40755	502/0
./Library/Frameworks/cisco-vpnclient.framework	40755	502/0
./Library/Frameworks/cisco-vpnclient.framework/Headers	40755	502/0
./Library/Frameworks/cisco-vpnclient.framework/Headers/vpnapi.h	100644	502/0	20265	1156317900

Delete any Cisco-specific files, (obviously don’t delete things like /Library/Frameworks) and then reboot to make sure there’s no stale daemons running or kernel extensions loaded.

Using the built-in VPN support doesn’t seem to cause any conflicts with VMware, I started the VPN while it was running and the virtual machines then had access to hosts on the remote network with no obvious problems. JFW, as they say.