Tag: imovie

Better safe than sorry

I can’t believe it has almost been 3 years since I wrote about my data storage and backup strategy! In that post, I mentioned near the end that I would be experimenting with online backup solution. To cut a long story short, I started with Mozy (crappy OS X client), then Carbonite (didn’t support external drive), and ended up using CrashPlan instead. It took a while (as in months) to back up all of my iMovie files, Aperture vault, plus other files totaling ~1.5TB to CrashPlan but it is all worth it because just over 2 weeks ago my Drobo started failing. The only reason I noticed it was failing is because I wanted to edit June’s NY Alt.NET meeting’s videos and iMovie hung on startup. Took me a while to pin it down to the Drobo/Hard Disks because I don’t have a lot of free time nowadays with 2 toddlers running around the apartment.

Turned out almost every file I tried to open on the Drobo failed with I/O error. And there didn’t seem to be a pattern as to which files are good and which are bad. Fortunately files I restored from CrashPlan seem to be fine so at least I have a good copy somewhere. After some more probing and waiting for Drobo customer support to response to my help ticket (not very helpful), I ended up purchasing the Synology DS1512+ NAS as a replacement.

Replacing my 2nd generation Drobo with a NAS has been on my plan for a while. Leah’s Macbook is getting full and some point in the future I will be replacing my 7 years old MacPro with a new machine, probably a laptop with SSD (Macbook Pro with Retina Display? Yum!) and it won’t make economic sense to store everything on the SSD or attached an external drive to a laptop. The Drobo failure just pushed the purchase decision up a few months.

The Hardware

The setup now consists of:

  • Synology DS1512+ with two Seagate Barracuda 7200 1.5TB and one Western Digital Caviar Green 1.5TB (repurposed from the Drobo) disks, using SHR (Synology Hybrid RAID) resulting in 2.68TB of storage, connected to the Cisco E3000 wi-fi router
  • MacPro now hardwired to the Cisco router via 50 feet long CAT5e cable
  • The NAS drive is mounted using AFP, with auto-mount setup

I’m still undecided on what to do with the Drobo enclosure and the remaining three 1TB drives. One of the drive will replace an old 500GB drive in the external drive that I use for mirroring my MacPro internal drive. I may create a RAID-0 setup for the MacPro using the remaining two 1TB drives. But then again, considering the amount of work involves I probably won’t.

The Restoration

CrashPlan

After setting up the NAS and created the volume which took less than an hour, the first thing I did is set up CrashPlan on the NAS itself. Since Synology NAS runs a variant of Linux (BusyBox), it is (relatively) easy to setup CrashPlan’s linux client and runs it headless. Relatively easy as in if you are familiar and comfortable with working in *nix terminal shell.

To administrate this headless client, I need to run the CrashPlan GUI client on my Mac and connect to the NAS via SSH. This turns out to be a little hacky. To make life a little easier, I make a copy of the GUI client, edited the necessary settings, and setup SSH local port forwarding. Even with this, I still need to open up a terminal, ssh into the NAS, then run the GUI client. I think it may be possible to script this process but I haven’t spent time to research that.

I have the CrashPlan+ Family Unlimited account and it is simply a matter of adding the Synology to CrashPlan as a new machine and started backing up files directly from the NAS.

iTunes

The first batch of files I restored were my iTunes library. The process to switch iTunes from using the Drobo to NAS was much simpler and more straightforward than I thought it would be. Simply change the iTunes library location in the Preferences. Clicked ‘Yes’ when it asked whether I want iTunes to move files to new location (this will only move some files). Then tried to play a file (any will do) and let iTunes figure out the file’s new location. Done. I didn’t use the library consolidate function because iTunes will try to copy the files to new location, which won’t work as the files were all corrupted on the Drobo.

I did use the consolidate function for Leah’s Macbook and the copy process took less than an hour over Gigabit ethernet. As the original files are left untouched at the old location, I’m planning to delete them to free up space after a week or so of usage.

Aperture

I keep the main Aperture library on local disk for speed with an Aperture Vault on the Drobo. Instead of restoring the vault, it will be much faster to just create a new vault on the NAS. Despite the fact I get pretty decent write speed to the NAS (~70Mbps vs. ~15Mbps with Drobo), it took a rather long time to create the vault. Probably due to the fact that Aperture vault consists of large numbers of both large files (images) and small files (delta edits, meta-data, etc.).

iMovie

This turns out the be the trickiest of them all for 2 reasons:

  1. Apple leverages the HFS+ file system and stores meta-data in the file resources. Since the NAS volume is formatted as ext4 and thus does not offer this capability, iMovie will not work properly with NAS.
  2. Apple decides to not even display the NAS volume in iMovie (most probably due to #1)

One solution is to create a disk image on the NAS that is formatted to HFS+, mount it in Finder, then move the iMovie event/project files to it. This definitely works for iMovie (and Final Cut Pro X) so I decided to keep that as an option and tried something less certain but less work. (I’m lazy, ok?)

Thanks to the power of Google, I found this undocumented terminal command that makes network volume show up in iMovie:

defaults write -app iMovie allowNV -bool true

I then moved the files to the NAS volume using iMovie just like I would with the Drobo. It took a long time for iMovie to start the file move operation but once it started the transfer speed was decent (~35Mbps). Much faster than the Drobo but no where near saturating the Gigabit ethernet.

I haven’t done a thorough check on the iMovie project yet. But the cursory check I’ve performed, it looks like this hack works. If and when Final Cut Pro X supports NAS, that will be a very compelling reason for me to upgrade.

Other files

Just straightforward CrashPlan restoration from my MacPro onto NAS volume. This will take a while as I’m only getting 9-10Mbps download speed from CrashPlan (my FiOS is 35Mbps both up and down). The restoration requires my MacPro to be running and with the heat during the day, I only run the machine during the evening so it will be at least a week before all the files are restored.

Lessons learnt

So what are the lessons from all this.

Drives fail, even redundant ones

One can only hope the RAID system will report the failure and continue to work while waiting for replacement disk. Drobo in this case did not do that. It just failed, epically and silently.

Online/offsite backup is necessary

Even with RAID system

Don’t be a cheapskate

Drink couple less Starbucks a month and you are protected.

Panic is avoidable

Because I knew I have a copy of the data safe with CrashPlan. Desperation will only come if you know there is a high chance you’ve lost your valuable data.

Fast internet connection means not a lot, if someone is throttling you

In this case, not CrashPlan but one of the backbone ISP between CrashPlan and me.

Will never buy Drobo again nor recommend it to anyone

I bought the Drobo because of its redundant feature. If a single disk failure corrupts data, I might as well go with a eSATA external drive and spare the expense. Even if Drobo sends me a replacement unit for free, there is no easy way to regain my trust again. How would I know if the new unit won’t fail silently like the last one?

3

Video recording equipment

I’ve been asked a few times last year about what equipment I use to record the NY Alt.NET meetings, most recently from Bill Campbell (@bill44077). So I thought I’ll share what I use and how I use them here for all to see (and googled).

Continue reading “Video recording equipment” →

0

iMovie needs batch export

Since I was a teenager, I have been fascinated by the process of video production but purely from the technical point of view. The complexity of capturing video, editing (the process not the artistic element), video effects, etc. drove my curiosity. A few years ago I worked with someone who also really into video as well. He captured videos for lots of family events and learnt to use professional tools and techniques. While I get really excited as I discussed the ins and outs of the equipment and setup my friend has, I was definitively unable to find a reason to take video for personal use. Capturing family memory is great but who actually would sit down and watch hours upon hours of family videos, apart from the family themselves?

It all changed when I started organising the NY ALT.NET events last September. We decided that we would video tape each meeting so members who were unable to attend would have a chance to participate, albeit in a passive form. As a result, I started learning to use iMovie ’08 as a tool to process approximately 2 hours of video from each monthly meeting.

When Apple rebuilt iMovie from the ground up for the ’08 version, lots of complains were raised because many features from iMovie ’06 were dropped. What the user gained, however, was a much simpler user interface that makes editing movies a snap. Without any preparation and little learning time, I was able to import, pick video clips, add title and transition, and export videos within minutes. Literally. And the new version iMovie ’09 added new features that were missing from iMovie ’06, such as Precision Editor, themes, more transitions, etc. I especially love the Precision Editor because I can easily control the entry and exit points of each clip with ease.

But, and there’s always a but, all these great time-saving features come to nothing when I need to export the video for people to consume. Because Apple is targeting the consumer market with iMovie (Final Cut Express for prosumer, and Final Cut Studio for pros), there is no facility to batch up video export.

Why is it important? Let’s look at the typical time I spent working on videos for a monthly meeting:

  1. ~2-2.5 hours importing from camcorder. This is fixed time because it is a MiniDV tape camcorder and the only way to get from tape to hard disk is to replay all the footage in realtime.
  2. ~20-30 minutes editing each part. I break down the 2 hours meeting into around 30 minutes parts (actual length depends on the nature stopping point of the conversation). I then change the opening credit, make some adjustment to the audio (boast volume) and video (colour correction).
  3. ~2 hours exporting each final video to hard disk for uploading. This result in a 640×480 H.264 QuickTime video file around 550MB.
  4. ~1 hours uploading to Vimeo.

Now, there is no way for me to multi-task step #1. There is only one camcorder and thus I can only do one import at a time. Not a big problem as I can do other things once the import has started.

Step #4 is a background process. Once I kicked it off in the browser, I can do other things on the computer. For example, step #2.

The problem is with step #3. iMovie does not have the ability to export video in the background. What it means is that once the exporting process has started, I can’t use iMovie to work on my next clip and thus save time. It would not be that bad if iMovie uses all the CPU power my computer has. My Mac Pro has two CPUs, each CPU has two cores, making it a total of four cores. That’s a lot of computing power. But iMovie can only take advantage of one CPU at a time! So instead of a reasonable export time of around an hour, I have to wait for two before I can work on the next clip.

Now, if iMovie is able to export in the background, then at least I can work on the next clip using one CPU while exporting uses the other CPU. Alternatively, if iMovie has the facility to batch up videos for export in a queue, then I can work on editing all the clips and export them in a single batch while I sleep, or do whatever. But iMovie has neither and so the process of producing four 30 minutes video clips basically takes up most of my weekend instead of a few hours. Granted most of the time is spent waiting but still…

Of course one can argue that background or batch exporting is a pro feature and I agree. But when should I sacrifice the ease of use of iMovie, learn a completely different editing paradigm (arguably more difficult to use), just so I can save a few hours each months when my hardware is more than capable? I am more than willing to pay extra for iMoviePlus or plug-ins to achieve what I need but unfortunately I don’t think either would be available any time soon.

11

Windy tennis

My Flip Video (I have the 2GB/60 min, black) arrived last week and I finally got the chance to play with it today. Unfortunately the weather didn't cooperate. Dan and I had to deal with 15-20 mph wind while trying to film our tennis swings. You can hear the wind noise in the background, it's kind of hard to miss! Here is a short clip of what we did:

You can see the full version (17:07) on Vimeo.
Transferring the video clips from the Flip to my Mac was a breeze. The Flip appears as a USB drive when plugged in so I can just copy the files across to my hard drive. Editing is a little tricky. The Flip uses a DivX MP4 encoder so the video files are not recognized by iMovie '08. After some Googling and experimentation, I decided to fork out some hard earned cash and got a license for QuickTime Pro. QTPro exports the clip to standard MP4 easily and quickly, utilizing all 4 of my MacPro cores. And editing in iMovie '08 pretty easy too. I was able to put this short clip together in about 5 minutes, even though I have no prior experience with iMovie '08 or video editing. It would be perfect if Apple fixes iMovie so that in the future it would just recognize the Flip video without any transcoding.

Read and post comments |
Send to a friend

0

Website Built with WordPress.com.

Up ↑

%d bloggers like this: