Bill Bumgarner

2004-12-5

Rsync: Backing up Music Purchased from iTunes Music Store

Obviously, backing up tracks purchased from iTunes Music Store on a regular basis is a very good idea (along with anything else you really don't want to lose).

I use an external hard drive for backups. I also like to burn all of my purchased music to a piece of optical media whenever I purchase new tracks (about once a month).

Now, I don't really need to backup all of my ripped CDs. The CDs are the backup. Given that Apple has consistent history of releasing better quality audio encoders fairly regularly, I'll need to re-encode my CD collection every couple of years anyway.

The purchased music is interleaved with non-purchased music. To back it up, I use rsync to copy the purchased music and only the purchased music to the external hard drive. From there, it is trivial to burn the copied files to CD or DVD.

The rsync command to synchronize the purchased music and only the purchased music is a bit tricky because rsync's include/exclude options are, uh, subtle.

Of note (yes, that is all one single command):

 rsync --block-size 32000 -e ssh \
--progress --partial -a --include '*/' --include '*.m4p' \
--exclude '*' \
'orange.local.:/Volumes/LargeData/Master\ Music\ Library/' \
/Volumes/ExterData/MusicBackup/Purchased\ Music

  • It works on any system that can resolve the name 'orange.local.' (the name of the machine with the music library) including on that machine itself. rsync is that smart.
  • The --block-size option increases the CRC block size to 32,000 bytes. This is several orders of magnitude larger than the default. Larger block sizes are much more efficient for large files.
  • --progress shows percentage transferred for each file
  • --partial will not delete files on the local end that aren't fully transferred. Very useful if you happen to be backing up across a slow or intermittent link or you simply want to hit ctrl-c in the middle and finish later.
  • -a is a catch-all option that enables a bunch of other options that are fairly optimal for archival purposes
  • The order of the --include and --exclude options are critical. The first includes "all directories", the second grabs all purchased music, and the third excludes everything not selected by the first two. Remove the first --include and rsync excludes the top level directory and does not recurse at all.
  • Annoyance: you'll end up with empty folders for all of the non-purchased content. Doesn't bother me that much, but I wish rsync had a "don't create empty folders" option.

Comment on this post [ so far] ... more like this: [Mac OS X, iTunes Music Store] ... topic exchange: [Mac OS X, iTunes Music Store]

Xcode, file references & build locations

Within Xcode projects, every file or resource has a reference that instructs Xcode how to find the file. This includes files, resources, build products, dynamic libraries and everything else in the project.

These references are both extremely powerful and quite easy to configure incorrectly. Any resource reference can be either via a relative or absolute path. If relative, it can be relative to one of several locations including the project, the enclosing group, or the build directory.

Now, Xcode also has a very useful feature that allows you to specify a location into which all build products and intermediate products will be written. First and foremost, this will eliminate the noise of the build directory from your workarea. No more "? build" noise from CVS or Subversion. If you point it to the fastest hard drive on your system, you will also gain the advantage of all the intermediate and final products being written to and read from that fast device. If you have a boatload of RAM, create a RAM drive and eliminate a huge chunk of disk I/O from the build process. If using FileVault, moving the build directories will eliminate the overhead of encrypting the files created during the build of the project (with the obvious security sacrifice).

Just make sure you get the resource references correct. In particular, If you have multiple targets and then have a copy files build phase that copies those targets' products into the product of some other target, make sure you use a build directory relative reference.

A project relative reference will work, but only on your system and only if your build directory never moves relative to your project directory.

And that is the real challenge to resource references. Misconfigured references will work on your machine the first time you set them, but will fail if used in an environment without exactly the same set of paths.

To help minimize the problem:

  • Use Xcode's preferences to set the intermediate and final build product directories to some place other than the project. Don't use the same path. I have often set the path to /tmp/build-bbum/intermediate and /tmp/build-bbum/products respectively. This also guarantees that the projects will be rebuilt after any reboot. Occasional clean rebuilds often seem to clean up weird issues.
  • Create a second user account on your system and use Fast User Switching to occasionally bop over, update a workarea and build the project. This will also rapidly pick up revision control problems, such as forgetting to add a resource to the repository.
  • Better yet, keep a workarea on a second machine or have a friend do an update and make sure things still build.

This came up because I had been playing with Growl and had checked out the source because I wanted to fix a few bugs. The CopyFiles build phase in the GrowlHelperApp were project relative, but referred to build products, thus assuming that the build directory is always in the same place relative to the project.

Comment on this post [ so far] ... more like this: [Mac OS X, Xcode] ... topic exchange: [Mac OS X, Xcode]