Better safe than sorry

I can’t believe it has almost been 3 years since I wrote about my data storage and backup strategy! In that post, I mentioned near the end that I would be experimenting with online backup solution. To cut a long story short, I started with Mozy (crappy OS X client), then Carbonite (didn’t support external drive), and ended up using CrashPlan instead. It took a while (as in months) to back up all of my iMovie files, Aperture vault, plus other files totaling ~1.5TB to CrashPlan but it is all worth it because just over 2 weeks ago my Drobo started failing. The only reason I noticed it was failing is because I wanted to edit June’s NY Alt.NET meeting’s videos and iMovie hung on startup. Took me a while to pin it down to the Drobo/Hard Disks because I don’t have a lot of free time nowadays with 2 toddlers running around the apartment.

Turned out almost every file I tried to open on the Drobo failed with I/O error. And there didn’t seem to be a pattern as to which files are good and which are bad. Fortunately files I restored from CrashPlan seem to be fine so at least I have a good copy somewhere. After some more probing and waiting for Drobo customer support to response to my help ticket (not very helpful), I ended up purchasing the Synology DS1512+ NAS as a replacement.

Replacing my 2nd generation Drobo with a NAS has been on my plan for a while. Leah’s Macbook is getting full and some point in the future I will be replacing my 7 years old MacPro with a new machine, probably a laptop with SSD (Macbook Pro with Retina Display? Yum!) and it won’t make economic sense to store everything on the SSD or attached an external drive to a laptop. The Drobo failure just pushed the purchase decision up a few months.

The Hardware

The setup now consists of:

  • Synology DS1512+ with two Seagate Barracuda 7200 1.5TB and one Western Digital Caviar Green 1.5TB (repurposed from the Drobo) disks, using SHR (Synology Hybrid RAID) resulting in 2.68TB of storage, connected to the Cisco E3000 wi-fi router
  • MacPro now hardwired to the Cisco router via 50 feet long CAT5e cable
  • The NAS drive is mounted using AFP, with auto-mount setup

I’m still undecided on what to do with the Drobo enclosure and the remaining three 1TB drives. One of the drive will replace an old 500GB drive in the external drive that I use for mirroring my MacPro internal drive. I may create a RAID-0 setup for the MacPro using the remaining two 1TB drives. But then again, considering the amount of work involves I probably won’t.

The Restoration

CrashPlan

After setting up the NAS and created the volume which took less than an hour, the first thing I did is set up CrashPlan on the NAS itself. Since Synology NAS runs a variant of Linux (BusyBox), it is (relatively) easy to setup CrashPlan’s linux client and runs it headless. Relatively easy as in if you are familiar and comfortable with working in *nix terminal shell.

To administrate this headless client, I need to run the CrashPlan GUI client on my Mac and connect to the NAS via SSH. This turns out to be a little hacky. To make life a little easier, I make a copy of the GUI client, edited the necessary settings, and setup SSH local port forwarding. Even with this, I still need to open up a terminal, ssh into the NAS, then run the GUI client. I think it may be possible to script this process but I haven’t spent time to research that.

I have the CrashPlan+ Family Unlimited account and it is simply a matter of adding the Synology to CrashPlan as a new machine and started backing up files directly from the NAS.

iTunes

The first batch of files I restored were my iTunes library. The process to switch iTunes from using the Drobo to NAS was much simpler and more straightforward than I thought it would be. Simply change the iTunes library location in the Preferences. Clicked ‘Yes’ when it asked whether I want iTunes to move files to new location (this will only move some files). Then tried to play a file (any will do) and let iTunes figure out the file’s new location. Done. I didn’t use the library consolidate function because iTunes will try to copy the files to new location, which won’t work as the files were all corrupted on the Drobo.

I did use the consolidate function for Leah’s Macbook and the copy process took less than an hour over Gigabit ethernet. As the original files are left untouched at the old location, I’m planning to delete them to free up space after a week or so of usage.

Aperture

I keep the main Aperture library on local disk for speed with an Aperture Vault on the Drobo. Instead of restoring the vault, it will be much faster to just create a new vault on the NAS. Despite the fact I get pretty decent write speed to the NAS (~70Mbps vs. ~15Mbps with Drobo), it took a rather long time to create the vault. Probably due to the fact that Aperture vault consists of large numbers of both large files (images) and small files (delta edits, meta-data, etc.).

iMovie

This turns out the be the trickiest of them all for 2 reasons:

  1. Apple leverages the HFS+ file system and stores meta-data in the file resources. Since the NAS volume is formatted as ext4 and thus does not offer this capability, iMovie will not work properly with NAS.
  2. Apple decides to not even display the NAS volume in iMovie (most probably due to #1)

One solution is to create a disk image on the NAS that is formatted to HFS+, mount it in Finder, then move the iMovie event/project files to it. This definitely works for iMovie (and Final Cut Pro X) so I decided to keep that as an option and tried something less certain but less work. (I’m lazy, ok?)

Thanks to the power of Google, I found this undocumented terminal command that makes network volume show up in iMovie:

defaults write -app iMovie allowNV -bool true

I then moved the files to the NAS volume using iMovie just like I would with the Drobo. It took a long time for iMovie to start the file move operation but once it started the transfer speed was decent (~35Mbps). Much faster than the Drobo but no where near saturating the Gigabit ethernet.

I haven’t done a thorough check on the iMovie project yet. But the cursory check I’ve performed, it looks like this hack works. If and when Final Cut Pro X supports NAS, that will be a very compelling reason for me to upgrade.

Other files

Just straightforward CrashPlan restoration from my MacPro onto NAS volume. This will take a while as I’m only getting 9-10Mbps download speed from CrashPlan (my FiOS is 35Mbps both up and down). The restoration requires my MacPro to be running and with the heat during the day, I only run the machine during the evening so it will be at least a week before all the files are restored.

Lessons learnt

So what are the lessons from all this.

Drives fail, even redundant ones

One can only hope the RAID system will report the failure and continue to work while waiting for replacement disk. Drobo in this case did not do that. It just failed, epically and silently.

Online/offsite backup is necessary

Even with RAID system

Don’t be a cheapskate

Drink couple less Starbucks a month and you are protected.

Panic is avoidable

Because I knew I have a copy of the data safe with CrashPlan. Desperation will only come if you know there is a high chance you’ve lost your valuable data.

Fast internet connection means not a lot, if someone is throttling you

In this case, not CrashPlan but one of the backbone ISP between CrashPlan and me.

Will never buy Drobo again nor recommend it to anyone

I bought the Drobo because of its redundant feature. If a single disk failure corrupts data, I might as well go with a eSATA external drive and spare the expense. Even if Drobo sends me a replacement unit for free, there is no easy way to regain my trust again. How would I know if the new unit won’t fail silently like the last one?

How safe do you feel?

Most people I know do not have any plan to backup their data. Somehow they put their trust to a piece of 3.5″ glass disc, spinning at 5400 times a second or more with a sharp metal spike fraction of a hair above it. Me? I like to spread my risk and backup my data, especially after suffering a total data lost about 12 years ago. As Alex Lindsay from the PixelCorps often says on podcasts, “Unless the data is stored in 3 different places, it doesn’t exist”. This may sound over the top but with all the online file sharing or storage services available it is actually pretty easy to have decent backup strategy with minimum cost. To achieve comprehensive backup coverage, it would definitely cost a few dollars.

Like computer security, any backup strategy should be constructed in layers. This applies to both time, location, and accessibility. Let me use my backup strategy as an example.

Continue reading “How safe do you feel?”

Tale of a data paranoid

Ever since I had my first hard disk failure about 11 years ago and lost a substantial portion of my data, I am more concerned with data back than most computer owners/users. Every new computer that I purchased since then had also included provision for data backup. I started out with magnetic tapes that while worked, took long time to back up even a moderate amount of data. Not to mention the longevity of the tapes were always in question. Even with the popularization of CD-ROM/RAM or even DVD-ROM/RAM, I never used it as a backup medium. Capacity is too low and unless I paid for top quality disks, longevity again was suspect.

In the last few years since I switched to OS X, I have been using external hard disks as a way to backup my data. The main catalyst are applications such as Carbon Copy Cloner and SuperDuper! which mirror a drive to another exactly. The advantage of unix-based OS X means that once my drive is cloned, I can actually boot up from the external drive and use it from there. Try doing that with a Windows machine!
Hard disk has its own issue though. With the advent of large capacity hard disk, that means I have to constantly purchase new external drive to accommodate any new internal drive, leaving a number of smaller and less useful disks around the apartment. And with the arrival of Time Machine in Leopard, now I need double the capacity of my internal drive so I can have both Time Machine and clone.
This is clearly a non-scalable solution.
I first heard of Drobo when I saw a video demo/interview of the Drobo founder by Robert Scoble. At the time I thought it was an interesting way to make RAID/NAS drive simpler to use, but I felt that the price was too steep for just an fancy enclosure. Fast-forward to last week when I recorded the first ALT.NET meetup and ended up with nearly 2 hours of DV video files. Suddenly the free space on my 250GB internal drive in my Mac Pro shrunk to almost nothing.
This time I look for a more scalable solution. RAID is an option but due to the stupidity of Apple it is not trivia to add software RAID or cheap to add hardware RAID. Also, the need to decide which type of RAID I want to setup seem to defeat the objective I want a Mac in the first place; to get work done instead of tinkering with software/hardware. So I give Drobo another look and found that while it may not be as integrated as RAID, the flexibility it offers more than offset that. The ability to easily add new drive(s) to expand my storage capacity means I don't have to decide on how large a disk I should purchase every time I need an upgrade. Now, I just buy a new drive, put it in to Drobo and that's it. No need to worry whether I should do RAID 0, RAID 1, or RAID 5, etc.
So how easy it is to get Drobo setup?

I purchased the generation 2 Drobo with 2 x 1TB drives package on the Drobo Store. After unpacking everything it was simply a matter of hooking up the power supply and the FW800 cable to the back of the Mac Pro, and pushed the two drives into the slots. Then I installed the Drobo Dashboard from the CD and formatted the Drobo "drive". One of the question the Dashboard asks is what size should Drobo drive be formatted at. I was given the choice of 1, 2, 4, 8, and 16TB. Logic or even common sense would say that with two 1TB drives I should pick 2TB option. But then I remembered Leo Laporte mentioned this on one of the MacBreak Weekly podcast that the best option is 16TB no matter how much storage is actually in the Drobo. Why? Because this mean I won't need to re-formatted the drive in the future or have multiple "drives" shows up in the OS when I put in more storage capacity.
Then it was a few minutes wait while the Dashboard "formats" the Drobo and rebooted it. Once that is done, the Drobo shows up in OS X as an external drive of 16TB. The Dashboard also offers to format the drive as HFS+ but somehow it didn't work for me. I had to use Disk Utility to partition and format but that was easy. (I create two partitions: one 250GB for Time Machine, and one that occupies the rest for data)
And that's it really. Total time from unpack to having a 16TB drive on my desktop? 30 minutes max. What is more time consuming is the data transfer from the internal drive to Drobo. The FW800 connection is really fast but it is not infinite! It still takes around 20 minutes to transfer my 40GB Windows XP VM while Time Machine is also backing up to the Drobo. Having said that, the connection is definitely fast enough to use Drobo as a primary drive. I have no problem running XP VM off Drobo directly. In fact, it is slightly smoother now because the VM is no longer on the internal drive, blocking OS I/O. The acid test would be to edit HD video directly from Drobo.

One thing I do take time and care is the transfer of my media files. Having lost iTunes metadata before, this time I look up the instruction and do it "properly". Here are the three links that explains how to transfer:

Read and post comments |
Send to a friend