Have MRTG monitor drives >2TB

Written by on 16th August 2012 in technology with 0 Comments

I ran into a problem where MRTG couldn’t graph disk space on arrays larger than 2TB.  This isn’t the fault of MRTG, but rather the SNMP OID I was using only went this high.  I found another set of OID’s that worked instead, but they were off by a multiple of 4.  In case anybody else runs into this problem, here is the code that worked for me.

Read More →

Running IPv6 when your ISP doesn’t yet support it [solved see comments]

Written by on 2nd July 2012 in technology with 1 Comment

I ran into an issue where a client wanted to run IPv6 internally for testing, but their ISP did not yet support it.  So we decided to use the FE00:: space and carved out a /48 for them to use.  We set each subnet internally to use a different /64, and thought all was well.

Until World IPv6 day that is (June 8th 2012).

Suddenly, websites (such as Google and Bing) take 30+ seconds to load.  I quickly realized that it was because the website supported IPv6, and DNS was returning a v6 address (which we didn’t have a route to).  I found out that there is no way to make Microsoft DNS server not return the v6 address (and only ipv4).  We can’t run a 6to4 tunnel, as we have nothing on the other side to turn it back int IPv6.  We couldn’t run a Teredo relay either for security reasons.

So far I haven’t found a solution to this, until your ISP supports IPv6 there is no good way for us to solve this.  It seems we should disable our “private ipv6” addresses we are using internally and let them fall back to link-local.  If anyone has a solution or idea please comment.

Read More →

ASA and MRTG VPN Bandwidth monitoring

Written by on 2nd July 2012 in technology with 0 Comments

I came across an apparent problem in an older Cisco ASA firewall and thought I would post about it.

Like many others, I wanted to monitor traffic across my site-to-site VPN’s.  I started searching and came across someone in a forum that said what the SNMP OID that MRTG needed to monitor were, so I tried:

snmpwalk -v 1 -c public 192.168.1.1 1.3.6.1.4.1.9.9.109.1.1.1.1.3.1

This showed that the VPN tunnel had a strange OID below it.

SNMPv2-SMI::enterprises.9.9.171.1.2.3.1.2.6340608 = INTEGER: 1
SNMPv2-SMI::enterprises.9.9.171.1.2.3.1.3.6340608 = STRING: "REMOTE_IP_WAS_HERE"
SNMPv2-SMI::enterprises.9.9.171.1.2.3.1.4.6340608 = Hex-STRING: FF FF FF FF
SNMPv2-SMI::enterprises.9.9.171.1.2.3.1.5.6340608 = ""
SNMPv2-SMI::enterprises.9.9.171.1.2.3.1.6.6340608 = INTEGER: 1
SNMPv2-SMI::enterprises.9.9.171.1.2.3.1.7.6340608 = STRING: "MY_PUBLIC_IP_WAS_HERE"
SNMPv2-SMI::enterprises.9.9.171.1.2.3.1.8.6340608 = Hex-STRING: AA AA AA AA
SNMPv2-SMI::enterprises.9.9.171.1.2.3.1.9.6340608 = ""
SNMPv2-SMI::enterprises.9.9.171.1.2.3.1.10.6340608 = INTEGER: 1
SNMPv2-SMI::enterprises.9.9.171.1.2.3.1.11.6340608 = INTEGER: 104
SNMPv2-SMI::enterprises.9.9.171.1.2.3.1.12.6340608 = INTEGER: 10
SNMPv2-SMI::enterprises.9.9.171.1.2.3.1.13.6340608 = INTEGER: 3
SNMPv2-SMI::enterprises.9.9.171.1.2.3.1.14.6340608 = INTEGER: 1
SNMPv2-SMI::enterprises.9.9.171.1.2.3.1.15.6340608 = INTEGER: 28800
SNMPv2-SMI::enterprises.9.9.171.1.2.3.1.16.6340608 = INTEGER: 297068300
SNMPv2-SMI::enterprises.9.9.171.1.2.3.1.17.6340608 = INTEGER: 2999483
SNMPv2-SMI::enterprises.9.9.171.1.2.3.1.18.6340608 = Counter32: 0
SNMPv2-SMI::enterprises.9.9.171.1.2.3.1.19.6340608 = Counter32: 1676
SNMPv2-SMI::enterprises.9.9.171.1.2.3.1.20.6340608 = Counter32: 11
SNMPv2-SMI::enterprises.9.9.171.1.2.3.1.21.6340608 = Counter32: 0
SNMPv2-SMI::enterprises.9.9.171.1.2.3.1.22.6340608 = Counter32: 0
SNMPv2-SMI::enterprises.9.9.171.1.2.3.1.23.6340608 = Counter32: 3
SNMPv2-SMI::enterprises.9.9.171.1.2.3.1.24.6340608 = Counter32: 0
SNMPv2-SMI::enterprises.9.9.171.1.2.3.1.25.6340608 = Counter32: 0
SNMPv2-SMI::enterprises.9.9.171.1.2.3.1.26.6340608 = Counter32: 0
SNMPv2-SMI::enterprises.9.9.171.1.2.3.1.27.6340608 = Counter32: 604
SNMPv2-SMI::enterprises.9.9.171.1.2.3.1.28.6340608 = Counter32: 10
SNMPv2-SMI::enterprises.9.9.171.1.2.3.1.29.6340608 = Counter32: 0
SNMPv2-SMI::enterprises.9.9.171.1.2.3.1.30.6340608 = Counter32: 0
SNMPv2-SMI::enterprises.9.9.171.1.2.3.1.31.6340608 = Counter32: 2
SNMPv2-SMI::enterprises.9.9.171.1.2.3.1.32.6340608 = Counter32: 0
SNMPv2-SMI::enterprises.9.9.171.1.2.3.1.33.6340608 = Counter32: 0
SNMPv2-SMI::enterprises.9.9.171.1.2.3.1.34.6340608 = Counter32: 0
SNMPv2-SMI::enterprises.9.9.171.1.2.3.1.35.6340608 = INTEGER: 1

Which didn’t quite make sense.  Furthur googling shows that the items I wanted were #19 and #27, and that the number 6340608 in this example was recreated as a new number every time the vpn reconnects, so I couldn’t just put these OID’s into my config like this.

Thankfully, someone else made a nice little script here which automatically updates the config.  But it turns out that this is not quite correct, as the ACTUAL script I want to monitor is:

#!/bin/sh
# Parses out the correct tunnel entry to monitor for the VPN
# Technique: http://www.unixsamurai.com/mrtg/asa-vpn.html
# SNMP OIDs: http://tools.cisco.com/Support/SNMP/do/BrowseOID.do?objectInput=cipSecTunnelEntry&translate=Translate
#
XYZ=`snmpwalk -v 1 -c public firewall 1.3.6.1.4.1.9.9.171.1.3.2.1 | grep "AD A6 47 2A" | awk -F'.' '{print $10}' | awk '{print $1}'`
echo VPN ID: $XYZ
snmpget -v1 -c public firewall 1.3.6.1.4.1.9.9.171.1.3.2.1.26.$XYZ | awk '{print $4}' > /tmp/xyz.txt
snmpget -v1 -c public firewall 1.3.6.1.4.1.9.9.171.1.3.2.1.39.$XYZ | awk '{print $4}' >> /tmp/xyz.txt

Notice that the OID’s are not the same as on that page.  MRTG was not actually graphing the correct tunnel.  Using these OID’s it counts all of the traffic, which is what I want.  Plus, these OID’s are updated more often (less than 5 minutes vs 15 for the others).

My mrtg.cfg files looks like this:

### This is the site to site VPN ###
# there is a script in /opt/mrtg/remote-vpn.sh that creates /tmp/xyz.txt

Target[asa_100]: `cat /tmp/xyz.txt`
MaxBytes[asa_100]: 100000000
Title[asa_100]: Traffic Analysis for VPN
PageTop[asa_100]: <h1>Traffic Analysis for VPN</h1>
Options[asa_100]: growright,nopercent

And now I can see the bandwidth across the VPN properly.

Read More →

NPS/RADIUS problem with VMware ESX 4.1 to 5.0 upgrade

Written by on 26th April 2012 in technology with 0 Comments

I encountered a strange problem so I thought I’d post about it.

We upgraded many VMware 4.1 ESX nodes to 5.0, and for the most part it went fine.  However after the upgrade, our wireless users couldn’t authenticate on the internal network, but the guest network still worked fine.  We were using a variety of Cisco access-points and they had worked perfectly fine with RADIUS auth before the upgrade.  I tracked down the error, the RADIUS server (now called Network Policy Server in Windows 2008) was throwing error 266.  This error would indicate it thought it was receiving a malformed packet!  We were using the E1000 NIC driver on the VM’s that hosted the NPS/RADIUS server, and had heard that it could cause problems.

I created a new VM with the VMXNET3 driver, and copied the NPS/RADIUS settings over to this new server, and set the access-points to authenticate to it instead.  This worked fine, so I changed the NIC driver on the old VM’s to VMXNET3, but this did not seem to solve the problem.  For now I just created new VM’s with the VMXNET3 driver and they are working fine, I am not sure what is different or why this doesn’t quite work properly, but if encounter this problem hopefully this fixes it for you.

Read More →

Optoma HD700X white lines fix

Written by on 25th April 2012 in technology with 0 Comments

I was given an Optoma HD700X projector for free, but it was broken.  About every 15 pixels across, there was a vertical white line.  I googled a bit and it seems that this happens fairly often on Optoma projectors (even shortly after they are bought).  It found that it was likely the main board that was in need of repair.

I called Optoma, and they said the main board was $270.  Since this was free, I didn’t want to pay to risk fixing it and have it not work, I’d rather put that much towards a new projector!  I noticed that if I pressed on the top behind the focus ring some of the white lines would go away, leading me to think it was a poor solder connection.  Since I already have a projector I thought I would try to fix this myself with a reflow.

Solder reflowing is simply heating the circuit board back up to the temperature it was at when it was created, so that the weakened solder joints liquefy and make contact again.  The HD700X is a rebranded HD65, so I followed partial disassembly instructions from here.  Then I put the board in a pie pan, then a turkey pan (to help dissipate the heat evenly), and put an inch of solder in with it instead of a pop-up timer, and put it in the oven.

Andy is totally just like Julia Child.

Then I set the oven for 550F (has to be above 270C) and waited.

After about 10 minutes, we noticed the solder had melted, the oven was at 465F and still heating up, so I turned it off and opened the door and we waited another 20min for it to cool.

Once it was cool I took it out, we reassembled the board, and it worked*! I was amazed it actually worked.

 

* Well it worked after I put it back together *correctly*, the power cable from underneath was backwards!

Read More →