Reducing latency on Equallogic storage with VMware vSphere

I had an older Equallogic PS6000E SAN, configured for RAID 6 that was attached to a couple of vSphere hosts. Being comprised of a bunch of 1TB 7200 RPM SATA disks, it wasn't exactly built for performance and I would often see it top out on IOPS for long periods of time in SAN HQ. After a bit of shuffling in our other datacenter, I freed up a PS6000XV SAN (600GB 15,000 RPM disks, in RAID 10) and decided to add it to the same pool in order to utilize the auto-tiering capabilities and boost performance of the SATA SAN. My problems with IOPS were solved, but read latency remained stubbornly high. As I spent more time looking at the graphs, I realized that, strangely, the latency was highest when the IOPS were lowest, which is the opposite of what you'd expect. Shouldn't requests be answered faster when there is less work to do?

I did a bit of Googling, and decided to re-read the Best Practices for VMware guide for Dell's Equallogic storage. Buried inside there are two very helpful tips, that I don't remember being there years ago when I set up those SANs for the first time.

The important bits are found on pages 9-11. The section on Delayed ACK describes EXACTLY what I was seeing, so I disabled it, and Large Receive Offload (LRO) for good measure. Note that this will require a reboot of your hosts, but that's what we have vMotion for, right?

As you can see in the graphs below, the improvements in my read latency were pretty stunning and instant. If you are experiencing high latency during periods of relatively low IOPS with your Equallogic SANs, then definitely give this a try.

Jungle Disk

I installed Jungle Disk on my home PC last night to use for backing up my personal data. I've been using Mozy for the last year, and was pretty happy with it, but since I don't back up a ton of data, I think Jungle Disk will be slightly cheaper, and has a few unique features. I chose RackSpace as the location to store my data, though Amazon's S3 service is the default. They both charge 15 cents per GB per month to host your data, but S3 charges an additional bit for every GB you transfer in or out, and RackSpace doesn't charge that. I'm not sure why it defaults to S3, when RackSpace is cheaper, but it does.

I'm very curious about their "Enterprise" offering. The rate they're charging is very very low, but at the same time, they aren't EMC, they aren't Symantec, and I wonder how many enterprises are really ready to trust their data to Jungle Disk...

I'm using the Desktop edition, and I like that it lets you also mount your backup space as a drive, though I was a bit dismayed at first that I had to schedule a particular time to do my backups, as I tend to leave my PC asleep. Going through the options, I saw it had the ability to wake my PC from sleep, as well as perform my backups at the next time the machine was on, which is how I actually prefer to run it. I also liked that it let me throttle bandwidth usage during certain hours of the day, so that I can turn it loose from midnight to 6:00 a.m., but it won't eat up all my bandwidth when I'm likely to be using it. I do wish that it would let me configure separate settings based on days of the week though, I'd be fine with giving up all of my bandwidth during weekdays too, but not on weekends.

Mozy

I've been using Mozy ( now owned by EMC) for the last couple of months for my home PC, and I'm pretty impressed. The initial backup takes a while, (14 hours for me) due to the slow uploads that are typical of residential cable internet, but my daily backups tend to take less than a minute, since the software is smart enough to back files up at the block level, rather than the file level. With a little Googling for coupons, it wound up costing me about $40 for a year of unlimited backup service, which seems reasonable to get all my files, photos, and tax records automatically out of the house daily to save me from disaster. I also looked at their MozyPro service, which is actually pretty decently priced as compared to the backup service I run at work. They charge $6.95 per server per month, plus 50 cents per GB per month. The major downside that I see is that you can only restore from the last 30 days. That's really all you need for disaster recovery, of course, but it doesn't protect you from the user who deletes something and doesn't notice it's gone until the next semester.

While I don't think Mozy is going to be replacing our campus backup system anytime soon, it might be a viable alternative for those areas who are doing workstation backups, as it eliminates the need to leave the machine on for the scheduled backup job (Mozy just runs whenever the user turns the box on). Workstations are cheaper than servers, at $3.95/month + 50cents/GB.