Notes from the server room

You should never be afraid of your...

CA bundle. 

Had a fun issue and an F5 LTM didn't automagically select the proper CA bundle, and another device on the long walk to egress didn't either.

According to the Geotrust Certificate Checker: "The issuer of the following certificate is not supported by the certificate installation checker" WHAT?!?  I used the proper cert and I installed the bundle, why wasn't it working??  What was even more confusing, was that Chrome and IE8 had no complaints.  Only FireFox and Geotrust had a problem with the certs.

Evidently, the LTM and the intermediate devices were not sending the proper chain certs to the client.  While the serial numbers on the intermediate certs matched, something was bad wrong in the middle.  My guess is that the F5 was automatically selecting the wrong chain.

Openssl's CLI toolset saved the day.

openssl verify -verbose -CAfile /path/to/CA-BUNDLE.crt  /path/to/CERT.crt

Swapping around the CA bundle in the above line, I was able to find the proper one.  I tied the SSL profile to this one specifically in the F5 and the other devices and it all 'just worked'.

ESX Madness

Scenario: One application server (Windows Server 2003, IIS)in a VM on ESX 3.5 update 2. Works fine, connects to DB on physical machine over a gigabit link. The addition of a second application server causes connections to the common database to fail under load testing, from both machines. Tried: Looked at IRQ usage in /proc/vmware/interrupts. All seems to be distributing across all cores as expected. Earlier versions of ESX have issues with IRQ usage... see http://www.tuxyturvy.com/blog/index.php?/archives/37-Troubleshooting-VMware-ESX-network-performance.html Checking and optimizing code... only made the problem less frequent. Problem isolated: The problem turned out to be linked to the nic on the VM. The flexible adapter was failing under load for whatever reason. Sadly, this is the only adapter available for Windows Server 2003 Standard 32-bit in ESX 3.5. The Fix: Shut down one of the VMs. Go into the virtual machine's settings change the OS type to Server 2003 enterprise 64-bit. Save and exit the settings window. Re-enter settings and add a network card to the VM. You will see that the e1000 is now available. After installing the e1000 change the OS type back to Server 2003 standard 32-bit and start the VM. In Windows, go to Device Manager and select Properties for the new network adapter. Shut off all of the offloading for TCP checksumming, etc and move your network settings to the new adapter, disabling the old one in Windows and disconnecting it in the ESX VM settings. In summation: Yes, it's a little like duct tape. But it worked for me. And if you are in a pinch and can not afford the downtime to thoroughly examine a flailing production machine, this may work for you too. If it doesn't work for you,

I am not responsible for data loss, downtime, etc... blah blah...

GoDaddy issues

Unfortunate customers of GoDaddy: If you have suddenly encountered abysmal performance (connect to Apache quickly, but takes 10+ seconds to begin receiving the web page) with your server in the last few days, do this: Look in your web server configs to see if you have name resolution enabled for your web logs. If you do, turn it off and see if that helps. GoDaddy provisioned one of my servers with 208.109.96.1 for resolution and the server is now unreachable. Call GoDaddy and get another DNS server to resolve against.

Advice on use of Nessus

FYI: Don't run Nessus against a machine hosted at Godaddy. They drop traffic for a few mins for any IP that appears to be portscanning. Aside from slowing down Nessus, any requests for anything from the originating IP address (are you running it from behind your corporate firewall?) will be dropped. May be able to mitigate by changing scan settings, however...