Vermont Veto Override

Congrats to Vermont on being the first state to legalize same-sex marriage via the legislature. I'll be curious to see what those against gay marriage on the grounds of "judicial activism" have to say now that the legislature has passed such a law, twice, even overriding the governor's veto. I bet Vermonters are feeling even more proud today than we Iowans did last week.

Mozy

I've been using Mozy ( now owned by EMC) for the last couple of months for my home PC, and I'm pretty impressed. The initial backup takes a while, (14 hours for me) due to the slow uploads that are typical of residential cable internet, but my daily backups tend to take less than a minute, since the software is smart enough to back files up at the block level, rather than the file level. With a little Googling for coupons, it wound up costing me about $40 for a year of unlimited backup service, which seems reasonable to get all my files, photos, and tax records automatically out of the house daily to save me from disaster. I also looked at their MozyPro service, which is actually pretty decently priced as compared to the backup service I run at work. They charge $6.95 per server per month, plus 50 cents per GB per month. The major downside that I see is that you can only restore from the last 30 days. That's really all you need for disaster recovery, of course, but it doesn't protect you from the user who deletes something and doesn't notice it's gone until the next semester.

While I don't think Mozy is going to be replacing our campus backup system anytime soon, it might be a viable alternative for those areas who are doing workstation backups, as it eliminates the need to leave the machine on for the scheduled backup job (Mozy just runs whenever the user turns the box on). Workstations are cheaper than servers, at $3.95/month + 50cents/GB.

Storing Student Stuff, Part II

If you haven't already, read Part I first. So, what am I considering for hardware? I'm considering several options, here is my homegrown solution, if I couldn't afford a Filer/NAS device:

Homegrown student storage system:

  • Dell PowerEdge 2970 Server
  • Dell PowerVault MD3000i iSCSI SAN
  • Dell PowerVault DL2000 disk-based backup system
  • I planned on filling that MD3000i with 15 1TB disks, and when you get to that level of storage, you seriously need to consider how you're going to back it up. Right now, we protect a little over 15TB with our current backup systems on campus, so I'd nearly be doubling the size if I fed this data into our existing system. Doing full backups over the network each week would take far too long, so I'd need to utilize synthetic backups in order to reduce the amount of network traffic and the size of the backup window required. To do that, I looked at the DL2000 system, with two shelves of disk storage attached to it. This is essentially a 2U Dell Server, with Windows and Backup Exec (or CommVault) installed, and shelves of disk attached to the back of the server to use for storing the backups. While we typically use NetBackup, I've fairly familiar with BackupExec, and knew that it could perform synthetics necessary to reduce network traffic. Purchasing a dedicated backup system for this project also made some sense because I could predict the annual cost of the backups quite easily. Our NetBackup system is run on a charge-back basis, based upon how many GB each "customer" on campus is protecting with the system, as that number would fluctuate wildly, so would backup charges. Using our NetBackup system would be cost-effective if the student storage system stayed small, say under 2 TB, but if it hit 10TB, it would be extremely expensive to feed into our other backup boxes.

    Now, if I go back to my list of possible protocols that would be useful in this project, which can I provide with this solution?

  • CIFS - This is easy, it's built into the Windows server I'd use
  • NFS - Should be possible to do in Windows, maybe with Services For UNIX, but we've really never done it before, and permissions can be VERY tricky. NetApp has a white paper laying out the problems associated with mapping NFS to CIFS permissions, and how they tackle it. I'm pessimistic about our ability to solve the problem nearly as well on our own.
  • WebDAV - Again, this is theoretically possible in Windows too, but we haven't done it before either. It may work easily, it may be extremely problematic for us.
  • HTTP/Web Access - I have yet to find a good piece of software for doing this, there are plenty for allowing web users to store files inside databases, I have yet to see a good one for sharing an NTFS file system through a web page.
  • SFTP - Again, should be possible, if a suitable bolt-on SFTP server can be found, though the ugly permissions monster may attack here too.
  • So what are the pros & cons of this approach?

    Pros:

  • It's relatively cheap
  • It's as flexible as I can make a Windows server, if I can find the right software packages and make them work.
  • It's based on Dell hardware, which we have a lot of experience with.
  • Cons:

  • It's not redundant, if a hardware problem strikes the server, it's going to be down for 4 hours while we get replacement parts
  • Implementing so many simultaneous file protocols would be extremely difficult. I have grave doubts about our ability to make all of that work well, and it would take a very long time to accomplish it.
  • "Only" scales to 45TB. You can hook three MD3000i boxes in tandem, but that's it. I don't realistically expect to need more in the lifetime of this project, but it is a limitation that's smaller than my other options.
  • Monthly downtime for Windows patches, as well as patches for various file serving protocols. I know a Filer will have better uptime, just because you don't have to install security patches monthly.
  • Disaster Recovery - In the event that the system is destroyed, purchasing a new one will take time, and restoring the backups to it could take up to a couple days. No big deal if the problem happens over Christmas break, but if it's the weekend before Finals, it would be bad.
  • Stay tuned for Part III next week.

    Storing Student Stuff, Part I

    For the last few months, I've been working on a project to provide some file storage space to all 12,500 of our students here at UNI. Though many (most?) other schools do provide storage space, we've never done it in a central organized fashion, though a few of the Colleges have dabbled with it over the years. I've met with the IT staff from the colleges to try to build a list of requirements, and met with students to ask what they want... Then, I tried to figure out what technology to use to meet those requirements. The obvious one is the typical file server, but one of the requirements I got back was high availability, which really translates to clustering, and clustering tends to translate to being a pain in the backside to set up and maintain, and the complexity it introduces can often turn into more downtime with configuration issues than the clustering prevents during hardware failures.

    So, that got me thinking about Filer technology, something I'd never worked with, but I knew that Joe Breu, a former sysadmin at our municipal cable utility who now works for Rackspace did. I asked him about it, and he suggested I look at a NetApp filer. I was quite impressed, but I've also been looking at NAS type devices from EMC and Sun as well.

    So how do all these choices stack up? What are our requirements? What is the best fit? What can we afford? Which of the requirements are "hard" requirements and which are just wishes? How much space does each student need? How big should we make the disk quotas? What percentage of students will use the service? What percentage of students will utilize their entire disk quota? How are we going to back all this storage up? What will backing this up cost? These are the questions I've been attempting to answer over the last few weeks, so I decided to blog it all to try and help keep it all straight.

    The first thing we tried to establish is what protocols were possible for accessing the data by students.

  • CIFS - This is a no brainer, all the Windows boxes will use it
  • NFS - Probably not a hard requirement, but definitely a nicety for the Linux/UNIX users and Macintoshes.
  • WebDAV - This would provide the ability for students to mount their files as a drive while off-campus or on their personal machines.
  • HTTP/Web Access - Some sort of web page for students to easily upload and download a file, for quick remote access
  • SFTP - So tech-savvy students can easily upload/download large numbers of documents remotely, yet not broadcast their passwords in plain-text like FTP would
  • So, obviously these things can be accomplished on a traditional server, but, trying to make them all work together seamlessly would be difficult. Having NFS & CIFS permissions coexist can be tricky, and I have yet to find a good web access application for sharing files from a filesystem on the web. There are plenty for storing files in a database, but I need to do it from a filesystem...

    Continue to Part II.

    Doc Fynskov

    Our family veterinarian, Dr. Paul Fynskov, died this week. Being a farm family with livestock, we saw a lot more of the vet than the average person does. Doc Fynskov was good-humored, kind, and really seemed to enjoy his work. I always enjoyed the days that we would work cattle as a kid, because I got to hang out with Doc all day. Every time I smell whatever antiseptic was used to clean his office, it instantly reminds me of him, and now it will make me miss him...