FileVault and EncFS on Mac OS X (Leopard)

In the past I have used FileVault in attempt to protect files in the event of my mac falling into dubious hands. I find it frustrating on a few counts:

• the root user can access your files while you are logged in


• it regularly asks to reclaim space when logging out, making logging out much slower

On my linux machine I am using EncFS, which is working out well. So I thought I would try and install EncFS via MacPorts. I ran into all the following issues:

• boost 1.34.1 still won't build on Mac OS X 10.5 Leopard


• MacFUSE 1.7 fails to build


• encfs: configure fails

Then I ended up with an error I couldn't find a solution for:

$ sudo port install encfs
---> Building encfs with target all
Error: Target org.macports.build returned: shell command " cd "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_ports_fuse_encfs/work/encfs-1.4.2" && make all " returned error 2
...
FileUtils.cpp:163: error: 'make_binary_object' was not declared in this scope
make[2]: *** [FileUtils.lo] Error 1
make[1]: *** [all-recursive] Error 1
make: *** [all] Error 2

Error: Status 1 encountered during processing.

So I uninstalled MacFuse in MacPorts and used the installer from the MacFuse homepage. After installing EncFS from here and a reboot, I had success.

UQ Student Infomation System Error

Tried to access my class timetable from the my.UQ website.

SmugMug

SmugMug doesn't have a free account, but it does have a 14 day trial.

As soon as you try and upload some photos, you are forced to categorise them in a gallery. I find this frustrating and instead prefer how flickr works - upload photos and organise using tags and/or sets (similar to galleries) afterwards. So as a workaround I created a 'Default' gallery, which contains all photos I don't explicitly put in other galleries.

As mentioned in my flickr post, I found viewing photos in a gallery more efficient than in a flickr set due to the thumbnails being present. There is also the flexibility to choose from a number of different layouts for each gallery. On the downside, a long description for a gallery takes up a lot of screen real estate at the top, pushing all the photos down. The navigation for some photo actions seemed a little inefficient as it requires clicking on the Photo Tools drop down box and then selecting the action to take.

The first step in improving privacy for family photo sharing is to create a SmugIsland. I set 'Hello world!' to 'No' and 'Hello smuggers!' to 'No - make me an island'. You can set a site-wide viewing password. However, this disables showing a map on your homepage. Google Maps is used and apparently Google requires that all usage of Maps be done without requiring a password. I was also confused by the privacy settings in a gallery. Even though I had setup a SmugIsland, a gallery privacy settings seemed to show that it was public. Supposedly this is correct and the gallery will inherit my SmugIsland settings.

To use maps requires getting a Google Maps key, which is very simple. From there on though, I had very little success with maps. It looks like you can only set the location of photos individually. Once configured I could see a map of my photos that I had set a location for, but a non-logged in user (e.g. family) just sees an empty box where the map should be. When I was logged in and clicked on a pin on the map, it zoomed in, but then showed no photos. This feature just does not seem to work with SmugIslands.

At this point I stopped further investigation as the negatives outweigh the positives. Looks like I will be using flickr.

Flickr

I already had a dummy yahoo id for dealing with freecycle, so signing up to trial flickr was simple. It didn't take long to set the default privacy level to viewable by family only, upload a few photos and add some tags and descriptions. The descriptions support hyperlinks, although you have to enter the HTML manually.

To set the geographical location on a photo you drag a pointer to the photo to a location on a map. It just so happened that one of my photos was taken at a place where the flickr map didn't have enough data, so you couldn't zoom in far enough to be accurate. Otherwise though, the mapping features seemed to work well.

I like the fact that uploading photos requires no categorisation and you can easily view all your photos (flickr calls it your photostream) from several different perspectives.

To share private photos with non flickr users, you use a guest pass. The help mentions that you can share your whole photostream using a guest pass, but the "Share This" link was not appearing on my photostream page. It seems there is a bug if all your photos are private. Making one photo public solved the problem. After generating the guest pass, the photo can be set back to the previous privacy level.

I find the site looks very clean and uncluttered, which I like. The navigation at the top and the expanded nav at the bottom was helpful, so getting around was easy. For flickr to be useful to share your photos you really need a Pro Account. It costs US$25/year and you get unlimited storage, which seems very reasonable.

Now for the things I don't like. When you click on a photo to view it, the resulting page has a widget that only shows thumbnails for the previous and next photos. You can go forwards or backwards in the thumbnails or click on browse to go back, but when viewing photos I found this quite cumbersome. SmugMug does this much better, by showing more thumbnails next to the main image. Viewing photos in flickr as a slideshow looks like a good solution though.

Users with a guest pass for the whole photostream (including private family photos) can see your sets and each photo's description, tags and camera data. However, they:

• cannot see the geographical location for, or find your private photos on the map


• cannot browse your private photos via your tags. The tags themselves are visible, but if all your photos are private, clicking on a tag takes you to a "no results found" page, which seems contradictory.


• cannot comment on your photos

In summary, flickr scores reasonably well against my requirements. Guest pass users have a very basic level of access. If they want more access to your photos, they need to sign up for their own account, which will cost them their time, but nothing financially. Then you can add them as a family contact.

Family Photo Sharing

Most of my extended family do not live near us and I would like a way to keep them more updated with what we are doing. I envisage one aspect of the solution will be sharing our photos with them more frequently than when we visit one another. With that in mind I started to consider what I need from a photo sharing site.

Given that the primary motiviator is sharing, an over-arching goal is convenience for family members. More specifically:

• They shouldn't have to sign up to the site if they don't want to. I would prefer to give them a url and possibly a password as a starting point. This way they only need to deal with a small part of the complexity of the overall site to achieve the main benefit. As an aside, I consider facebook an example of what not to do in this regard.


• Can view all photo details, such as date, tags/keywords, description, geographical location, camera details


• Can browse by date, album, tags/keywords and a map.
  
  I am using the term album to refer to an arbitary grouping of photos that I specify, e.g. for a holiday, birthday party, etc. Ideally the album would also have a description of its own.


• Original photo is downloadable


• Can make comments on photos


• Flexibility to receive notifications of new photos by a mechanism they understand (e.g. RSS, email), if they wish.

From my perspective:

• Convenience. The easier it is to upload photos and describe them, the more likely I will actually do it.


• Security. I wish to restrict my photos to people I choose and then everthing is read-only for them, except for making comments.


• Photo metadata. Tags/keywords, geotagging and a textual description that supports basic HTML such as hyperlinks.


• Retain ownership of all my content


• Some sort of group/community feature might be useful. For example if we are on holiday with other family members (that choose to use the same photo site) then being able to group our photos together for that time might be handy.


• Export/backup. A tool or API that enables photos and metatdata to be exported for backup.

In building my shortlist of sites to try I stumbled upon 45 Photo Sharing Sites.

I discarded photobucket due to its limits and sharing in picasa web albums looked confusing. I am trying out flickr and smugmug.

Converting a CD to FLAC Files (Mac OS X)

I have the following software installed:

• xAct 1.64


• Max 0.8.1


• FLAC tools for OSX 1.1.4


• md5deep via MacPorts

The following process would be significantly simpler if one app provided both a decent rip log and metadata lookup.

There is no point being concerned with the quality of the audio unless you know if there were errors introduced in ripping the data off the CD. xAct shines here. Max does not provide enough details, although there is a ticket open with the developer to fix that.

Max does well with metadata. xAct is poor.

The process
$TMP and $GIT_REPO represent file paths, e.g. $TMP=$HOME/tmp

1. Rip CD using xAct, saving files to $TMP/xAct. [1]


2. Save xAct output to $TMP/xAct/xAct.rip.log


3. Check the log for errors. After the first section of the log file, search for each occurence of 'track' to quickly find the right spots to check.


4. Open Max. It will automatically detect the CD and pop up a window. Check the metadata and fix/enter it if necessary.


5. Download album art. Press Command-D in Max. It will automatically search. Choose one of the listed images (assuming some are found). [2]


6. Rip to FLAC using Max (I have set my output directory to $TMP)


7. $ mkdir $TMP/max-rip
$ mv $TMP/[Artist]/[Album]/*.flac $TMP/max-rip


8. Convert $TMP/xAct/*.wav to FLAC using Max.


9. $ mv $TMP/*.flac [Artist]/[Album]


10. Ensure that all files in [Artist]/[Album] have the same names as those in max-rip. Assume max-rip as correct - we set the metadata in step 4. The following should identify if there are any differences:
    
    $ cd $TMP/[Artist]/[Album]
$ ls -1 ../../max-rip > /tmp/ls.txt; ls -1 | diff /tmp/ls.txt -; rm /tmp/ls.txt


11. Generate FLAC fingerprints:
    
    $ metaflac --show-md5sum *.flac > ffp.txt
$ mv ffp.txt $GIT_REPO/flac fingerprints/[Artist] - [Album].ffp.txt


12. $ mv $TMP/xAct/xAct.rip.log $GIT_REPO/log/[Artist] - [Album].xAct.rip.log


13. Transfer tags from the files ripped with Max. I use the following script:
    

    find . -name "*.flac" | while read FILE
    do
      FIXED=`echo $FILE | sed 's/ /\\ /g'`
      metaflac --export-tags-to=- "../../max-rip/$FIXED" | metaflac --import-tags-from=- "${FILE}"
    done

14. Transfer album art (if it was found and used in step 5) from the files ripped with Max. I use the following script:
    

    metaflac --export-picture-to=cover.jpg ../../max-rip/01*
    ls -lh cover.jpg
    echo "Adding album art ..."
    find . -name "*.flac" -exec metaflac --import-picture-from=cover.jpg {} \;
    rm cover.jpg

15. Move $TMP/[Artist]/[Album] to music library


16. rm -rf $TMP/xAct $TMP/max-rip


17. In the root directory of the music library:
    $ md5deep -rl * | sort > $GIT_REPO/md5deep.txt


18. Check the $GIT_REPO changes and commit

[1] See FLAC Encoding Guide for mac for a details on using xAct.

[2] At this stage I do not know the correct way to handle album art, as the players have a variety of ways of dealing with it. For the moment, I have chosen to store it in the metadata of each file.

Music Management

After going to all the trouble of ripping and encoding my CD's to a lossless format, I want to:

• Ensure integrity of the music library, i.e. at any point be able to validate that all the files exist, their contents haven't changed and that there are no extra files.


• Have a recovery strategy should there be a problem with the files.

Ideally, I would satisfy these requirements by placing all the music in a Distributed VCS and storing a master copy somewhere like S3. Unfortunately there are a couple of problems:

• I tried out Git, but after the initial commit of a music file, the repository storage space on the filesystem took up twice the size of the music file. Furthermore, changing metadata such as fixing a spelling mistake in the track name and committing increases the repository by the full size of the file again. I assume this is because the files are binary and already compressed. I didn't try out Mercurial, but I expect it will be the same.


• The music files are already large, even without the extra overhead of the previous point and the data transfer costs here in Australia are just too high.

My current solution:

• Store the music library on a removable drive on the Mac at home.


• Keep a copy of the music library on my computer at work by either periodically taking in the removable drive and using rsync or copying newer music onto a USB drive if physical space is at a premium, such as when cycling.


• Put checksums of the files in a Git repository stored on both machines. I can then verify the integrity of a music library at any time. Currently I use md5deep because it can recursively process a directory tree and is available for both linux and Mac OS X. The default md5 program on the Mac does not seem to have the same feature set as md5sum on linux.


• I also store FLAC fingerprints in the Git repository. FLAC files store a checksum of the uncompressed audio in the metadata and various tools, such as xAct on the Mac, can verify the file against that. I am not sure how useful storing the fingerprints is, but I can think of a few unlikely situations where it might be helpful, plus it is small and easy to generate anyway.

To verify a music library, I do:

$ cd $MUSIC_LIBRARY
$ md5deep -rl * | sort | diff $GIT_REPO/md5deep.txt -

where $MUSIC_LIBRARY and $GIT_REPO represent appropriate file paths.

I originally tried the matching feature of md5deep instead:

$ cd $MUSIC_LIBRARY
$ md5deep -rX $GIT_REPO/md5deep.txt *

However this does not catch the case where a file has been deleted in the music library but is still present in the Git checksum file.

FLAC Players on Mac OS X

iTunes doesn't play FLAC files by default, which is not suprising given Apple has kept ALAC proprietry and FLAC is an open competitor.

Cog is a free, open source audio player that supports FLAC. It works, but is fairly minimalistic. I couldn't find any way to manage playlists in the UI (version 0.07), although the Help indicates it supports M3U and PLS. There is no real concept of a library of music - just the files and folders as organised on the filesystem.

Songbird is a free media player built on top of the Mozilla stack. I think the idea is to access media from the web and your own local media in the one tool. Apparently it has FLAC playback issues. It also doesn't read album artwork from metadata yet.

Fluke is small utility that enables FLAC files to be played in iTunes. The version I tried (0.11) was a little cumbersome to use as you need to import the FLAC files with Fluke, which I found to be a slow process. Secondly, the track number and album art from the FLAC metadata did not show up in iTunes.

Play is a free audio player. It shows your music as a library that can be browsed via different attributes, has playlists and keyboard shortcuts. No cover flow or album artwork though, but for me that is icing, rather than a necessity.

I am trying Play for now and will give Songbird another go once it is more reliable.

Scoodi vs Freecycle

Was asked about this recently. Three reasons I prefer Scoodi over Freecycle:

1. With Freecycle I drown under the volume of email. With Scoodi, I have more options to tailor the way I hear about new listings (finer granularity on my geographical area, email, RSS, etc). I find RSS more useful than email for notifications like this.


2. Freecycle doesn't work well if there are no takers at the time an item is posted to the mailing list. Scoodi provides a solution for this case. It was designed to be a big searchable/browsable inventory of available stuff near you.


3. Scoodi has free and 'for sale' stuff on the same site, where as Freecycle is only for free items. I find this convenient, as an item is still being recycled/re-used whether it changed hands for free or if there was money involved.

Lossless Audio Formats

I like to rip my CD's to a lossless format so I have:

• a backup if they get damaged, lost or stolen


• the freedom to decide (and change my mind) on the audio quality/size trade-offs. I can listen to the lossless version on devices that have enough space and encode into a lossy format where space is limited.

In the past I have used Apple Lossless (ALAC), because iTunes was available on all the computers I used frequently and it was convenient. However, Apple has not publicly released an encoder, so iTunes is the only choice for encoding to ALAC. I do not wish to be restricted like this (tis a bit Microsoft-ish) as managing a music collection can take a fair bit of effort and I don't know what platforms I will use in the future.

I have started to try out FLAC. It is an open source format that seems to compare well with other lossless options.

Also looking at the ripping process, managing tags and album artwork, converting between formats, etc. There are a lot of issues to solve.