MATLAB findpeaks function

The findpeaks function in the MATLAB (verion 7.6.0.324) Signal Processing Toolbox finds local maxima or peaks in a vector.

>> v = [0 3 0 0 3 2];

>> [pks, locs] = findpeaks(v)
pks =
     3     3

locs =
     2     5

locs contains the locations or indices of the peaks in v (MATLAB indexing starts at 1).

For each column of a 256 x 5188 matrix we were calculating a simple moving average (smooth function) and then finding the peaks and it was taking a long time, approximately 19.6 seconds. We whipped up our own findpeaks function based on the following two lines:

>> s=v(2:end)-v(1:end-1);
>> loc=find((s(1:end-1)>0) & (s(2:end)<0)) + 1
loc =
     2     5

This calculates the pairwise differences between elements, i.e. the slopes and then looks for an up slope immediately followed by a down slope. Pretty simple and now the 5188 matrix columns are processed in approximately 1.3 seconds.

Upgraded to Velocity Gold, not

I received an email from Velocity rewards notifying me of a free upgrade to Gold membership. It was quickly followed up by another email containing the following.



Was it a human data input error or programming error?

Just say No!

Most software projects produce a mess and are a mess to manage. Projects that make it to production generally go on to a have a lifetime measured in years. While they are alive new projects are started, accumulating more mess. If the internal IT budget/group increases there is capacity to create an even bigger mess.

An obvious way to reduce the total mess, is for each project to be less messy. Unfortunately, there is little hope to do this at present due to the inherent nature of (current) organisations and the immaturity of software development in general. Another way is to do less software projects and make those smaller.

The business orientated technical managers I know seem to understand this at some level and are dealing with it by purchasing more off-the-shelf or outsourcing. This makes some of the costs of the mess more visible and attributable to (hopeful) benefits. However, it doesn't really address how poor we are at actually figuring out requirements, transforming them into usable, executable software and adapting to new requirements over time. With outsourcing, the mess is now less visible, but as a business relies on the systems and data, it will still experience similar effects from the mess over time.

From a purely technical perspective, does the world really need another bloated Java, Spring, Hibernate, Oracle, JUnit abomination application? Feel free to substitute other languages and libraries, such as Ruby and Rails, Groovy, etc.

Reduce the software mess. Do less software projects. Just say no!

Mac Flickr client

I am looking for a native mac application that provides an easy interface to my Flickr account. Requirements:

• Uploading - need to easily filter out the good photos from the camera, upload and add metadata.


• Browse photos in photostream, or by sets, tags, etc


• Edit metadata


• Save photos locally

From what I have briefly read, iPhoto 09 integration with Flickr is very limited. I use Flickr as the primary repository for my photos and iPhoto isn't intended to be a passive Flickr client. I found the following other candidates:

• Photonic Really nice app. Demo is free but expires after 14 days. Registration is US$25. Can browse photostream and sets, no coverflow view. Can search by tags, but you need to know the tag names. Double-click a photo to view a larger version. Doesn't show descriptions and no metadata editing or slideshow. However, there is a convenient link to a photo's Flickr page to do these. Apart from assistance with tags, the other major missing feature is the ability to download a photo or set for use as a desktop background or with the screen saver.


• FlickrEdit Java app that looks horrible with a clunky UI. However you can backup and download photos and sets. Apparently it saves some metadata in the IPTC header of downloaded files, but I can't see it with the mdls command.


• DeskLickr Set the Desktop background to a Flickr photo. You can point it to your own account, but it only uses your public photos.


• Cooliris Firefox add-on A 3D wall for viewing images. Works with Flickr, so if you open it on the first page of your photostream or on a Set page, you can view all those photos. My head is spinning from the animation, so it would be nice to turn that right down. More keyboard keyboard control (photo selection and paging) would make this really good.


• Picture Sync Essentially just an uploader. Didn't seem very intuitive to me.


• FotoViewr Webapp that provides 3D viewing of Flickr photos (like coverflow). Only works with public photos.


• Fotobounce Windows only now, with a mac version due this year.


• 1001 Didn't try this out.


• Flickr Finder A very simple Finder-like column view. Content layout seems to be broken on Snow Leopard.


• Photo Grabbr Didn't try it. Suspect it is public photos only.

None of the apps did uploading really well. I am using the standard mac Image Capture application to get photos off the camera. Finder Quick Look in Snow Leopard provides an easy way to look through photos and delete unwanted ones. The Flickr web site uploader and metadata editing work well enough.

Photonic is the easy winner for viewing photos. If it had better tag support and desktop/screensaver integration or the ability to download original images, then the license fee would be more easily justified.

I have made a Flickr set of my favourite photos for use in my screen saver. FlickrEdit is clunky, but I can use it to quickly download the whole set. This is all I use it for. It also includes an incremental backup feature (I need a backup solution too), but I haven't investigated that yet.

QOTD

Teaching for understanding didn't offset the destructive effects of telling them how to get the answer. Any step-by-step instruction in how to solve such problems put learners at a disadvantage; the absence of such instruction was required for them to understand.

-- Alfie Kohn

From: Education's Rotten Apples

Apple store error

Just wanted to find out the status of my Snow Leopard order. Nice fonts though.

R Resources

Been learning to use R for a stats subject. These links were useful.

• Quick-R


• R programming for those coming from other languages

Other things I have stumbled across along the way.

• R - Resources


• Rattle GUI for data mining in R


• SNA in R Talk, Updated with [Better] Video


• Revolutions blog

Update 1/11/2009
For completeness I should also have included An Introduction to R that comes bundled with R, as well as the official R Wiki. Also discovered RSeek.org as well as this MATLAB / R Reference.

FsCheck xUnit integration

I am using xUnit.net at work on an F# project. I wanted to incorporate FsCheck and check properties via xUnit.net, but there is no built-in integration. I tried this from Matthew Podwysocki, but it is out of date with respect to the versions I am using (xUnit.net 1.5, FsCheck 0.6.1).

EDIT 28/09/2009: Here is the new and improved version based on Kurt's comment.

module FsCheckXunit

open FsCheck
open FsCheck.Runner 
open Xunit 
   
let xUnitRunner = 
    { new IRunner with 
        member x.OnArguments(_,_,_) = ()  
        member x.OnShrink(_,_) = ()
        member x.OnFinished(name, result) = 
            match result with 
                | True data -> Assert.True(true)
                | _ -> Assert.True(false, testFinishedToString name result) 
        } 
        
let config = {quick with Runner = xUnitRunner}

Below is the first version I adapted from Matthew's post and the FsCheck source code.

module FsCheckXunit

open FsCheck
open FsCheck.Runner 
open Xunit 
   
let xUnitRunner = 
    { new IRunner with 
        member x.OnArguments(_,_,_) = ()  
        member x.OnShrink(_,_) = ()
        member x.OnFinished(name, result) = 
            match result with 
                | True data ->
                    Assert.True(true) 
                    data.Stamps |> Seq.iter (fun x -> printfn "%d - %A" (fst x) (snd x)) 
                | False (data,_,args,Exception e,_) ->
                    Assert.True(false, sprintf "%s - Falsifiable after %i tests (%i shrinks): %A with exception %O"
                        name data.NumberOfTests data.NumberOfShrinks args e) 
                | False (data,_,args,Timeout i,_) ->
                    Assert.True(false, sprintf "%s - Timeout of %i milliseconds exceeded after %i tests (%i shrinks): %A"
                        name i data.NumberOfTests data.NumberOfShrinks args) 
                | False (data,_,args,_,_) ->
                    Assert.True(false, sprintf "%s - Falsifiable after %i tests (%i shrinks): %A"
                        name data.NumberOfTests data.NumberOfShrinks args) 
                | Exhausted data ->  Assert.True(false, sprintf "Exhausted after %d tests" (data.NumberOfTests) ) 
        }
    
let config = {quick with Runner = xUnitRunner}

Erik Meijer's influence on mainstream programming

Many people are trying to influence those in their field. In Confessions Of A Used Programming Language Salesman (Getting The Masses Hooked On Haskell) Erik Meijer talks about his journey and how he achieved massive influence on mainstream programming.

For many years I had been fruitlessly trying to sell functional programming and Haskell to solve real world problems such as scripting and data-intensive three-tier distributed web applications. The lack of widespread adoption of Haskell is a real pity. Functional programming concepts are key to curing many of the headaches that plague the majority of programmers, who today are forced to use imperative languages. If the mountain won’t come to Mohammed, Mohammed must go to the mountain, and so I left academia to join industry. Instead of trying to convince imperative programmers to forget everything they already know and learn something completely new, I decided to infuse existing imperative object-oriented programming languages with functional programming features. As a result, functional programming has finally reached the masses, except that it is called Visual Basic 9 instead of Haskell 98.

Infiltration is the key! Obviously Erik is a very smart guy who has put in the hard yards and knows his stuff. However to achieve major impact required joining an influential organisation and working on the inside, rather than an external activist or confrontational approach.

The whole paper is worth reading, although a large portion is naturally Microsoft orientated on .NET language/CLR features. Below are just a few interesting quotes.

From section 5.2 Standard Query Operators

It is quite surprising that Haskell is one of the very few languages that allows higher-kinded type variables. If, on the term level, parametrizing over functions is useful, doing the same on the level of types sounds like an obvious thing to do. As far as we know Scala is the only other language besides Haskell that also support higher-kinded types.

From section 6.2.3 Contracts

The static type systems of most contemporary programming languages are actually not that expressive at all. They only allow developers to specify the most superficial aspects of the contract between caller and callee of their code.
...
From a program-specification point of view, most programs are extremely dynamically typed!

It is a real shame Erik decided to join Microsoft rather than Sun or IBM. Although I guess he probably wouldn't have had the same level of influence on the JVM and associated languages.

Learning - context and details

My mathematics lecturers often try and introduce a new topic with a motivating example. I think this is a great idea. I am a slow learner and find that an initial concrete example is the best way to approach a new abstract concept. I realise that this can sometimes pose challenges further on, i.e. in generalising from the concrete to the abstract, but this way seems to work the best for me.

Unfortunately, there are many times that the motivating example is not that motivating. I view teaching as often building a bridge from the student's current position to where you want to take them. To achieve that, requires providing an initial high-level context in terms that the student can relate to.

As an example, earlier this year I studied introductory linear algebra that included eigenvectors. Kudos to the lecturer for trying to motivate with a modern example (search engine page rank). Even though I am a professional programmer, this example didn't really work for me and I finished the subject with only a vague idea about them.

Recently I read this article which introduced them with simple physical examples. The second paragraph of the current wikipedia page nicely related it back to matrices and vectors and it all fell into place.

Some mathematical ideas are represented as a sequence of steps, e.g. calculating Maximum Likelihood Estimators in statistics. Once a good motivating example is done to provide context, things can proceed between two extremes.

1. Present high-level steps and then dig down into the details of each step


2. Work through the details of each individual step and bring it all together at the end

In the latter approach, I sometimes miss the high-level steps if I get lost in all the detailed algebra and reasoning underlying each step. Can't see the forest for the trees. I prefer the former approach as it provides more purpose and is therefore more motivating. It also makes it easier to isolate difficult sections that can be returned to later, so as not to get bogged down and lost for the remainder of the discussion/lecture.

I wish I had utilised these ideas better in my Intro to Functional Programming talk a few weeks ago.

Re: Static Typing and Functional Languages

I unsuccessfully tried to post a comment on Static Typing and Functional Languages.

First thing to note, is that Functional Java ‘looks’ so ugly, because Java is not naturally suited to expressing these abstractions.

Secondly, type inference appeared in the 1970s, well before Java existed.

Thirdly, the assumption that functional programming implies static typing is difficult. Type system research has been largely motivated by a number of factors over the years, including compiler optimisations for program performance and theorem proving.

GMail down

QOTD

‘CL is great and Blub is crap’ has not worked as a strategy for selling Common Lisp. People invest years of their life learning Blub and don’t want to hear this message. A strategy of infiltation is better.

-- Mark Tarver

From The Next Lisp: Back to the Future.

X-bit labs error

Just wanted to read the story about Nokia's new netbook.

QOTD

Whether it's being done in honest ignorance, blind obedience, or cynical exploitation of the market, the result is the same: our ability to envision new solutions to the latest challenges is stunted by a dependence on market-driven and market-compatible answers.

-- Douglas Rushkoff

From Economics is not Natural Science. An interesting article arguing that our current economy is derived from an archaic set of man-made rules designed to serve particular interests. Rushkoff calls for consideration of alternative models (not based on scarcity, competition and centralisation) rather than assuming the status quo.

LaTex resources

A few weeks ago I decided to bite the bullet and learn LaTeX. To make it happen I foolishly decided to type up my stats assignment (due yesterday) in it. Made for a very busy week, with my Bris FP talk, work, etc.

Anyway the following sites were useful.

• Getting to grips with LaTeX


• Getting Started with LaTeX


• Art of Problem Solving

Intro to functional programming talk

Last night I spoke at the Brisbane Functional Programming Group. Tried to scribble on the whiteboard a bit to avoid "death by slides". The objective was to help imperative programmers new to FP become familiar with some of the core concepts, so I tried to go at a slow pace. I think it went for about 1 hour 20 minutes.

The first part of the talk covered history and the landscape of popular functional languages. The rest was on functions, partial function application, algebraic data types, pattern matching and tuples. I used Haskell syntax, but the concepts are transferable to other languages such as Scala and F#.

Zoho Writer LaTeX equation editor

I think the LaTeX support in Zoho Writer's equation editor is a fantastic idea. Although, I don't know what would happen to the LaTeX source when exporting for backup purposes.

UPDATE: Document can be exported as LaTeX.

I ran into the following issues with the equation editor.

• The sigma representing a sum is tipped backwards at an angle when used in the denominator of a fraction. Changing to italics seems to straighten it up.


• I couldn't figure out how to change the font.


• When exporting to PDF, the equations looked a little blurry compared to text.

So I will stick with TexShop on the Mac for the moment. However, the Zoho editor is very useful for finding the LaTeX commands for mathematical symbols.

Strong Inference, Science (1964)

Strong Inference. Certain systematic methods of scientific thinking may produce much more rapid progress than others.

Strong inference consists of applying the following steps to every problem in science, formally and explicitly and regularly:

1. Devising alternative hypotheses


2. Devising a crucial experiment (or several of them), with alternative possible outcomes, each of which will, as nearly as possible, exclude one or more of the hypotheses


3. Carrying out the experiment so as to get a clean result

Rinse and repeat to form a "conditional inductive tree" or decision tree. Write the tree out.

Or, as the philospher Karl Popper says today, there is no such thing as proof in science -- because some later alternative explanation may be as good or better -- so that science advances only by disproofs. There is no point in making hypotheses that are not falsifiable, because such hypotheses do not say anything: "it must be possible for an empirical scientific system to be refuted by experience".

Form multiple working hypotheses, so as not to get too emotionally attached to any particular one.

Scala version of F# orElse getOrElse functions post

Just a quick comparison of F# Option orElse getOrElse functions in Scala. Given orElse and getOrElse are already defined in Scala, the equivalent code is:

def f(s:List[(Int, Int)]) = {
  def tf(g:Int => Boolean) = s.find {case (_,x) => g(x)}
  tf(_ < 0) orElse tf(_ == 0) map(_._1) getOrElse 10
}

The argument to orElse is defined as lazy in the function declaration via : => Option[B] rather than :Option[B]. However in the F# version, the actual argument type is changed from 'a option to 'a option Lazy.

Why I am Not a Professor

I currently study at one uni and work at another (not in an academic capacity though). Unfortunately Why I am Not a Professor OR The Decline and Fall of the British University seems to aptly describe things.

To the claim that academia is living a lie, I would add that many in industry and government are doing just the same, except with different situational parameters.

F# Option orElse getOrElse functions

Consider the following function that takes a Sequence of (pairs) int * int.

let f s =
  let o = Seq.tryFind (fun (_, x) -> x < 0) s
  if Option.isSome o then
    Option.get o |> fst 
  else
    let p = Seq.tryFind (fun (_, x) -> x = 0) s
    if Option.isSome p then Option.get p |> fst
    else 10

Essentially I am looking for the first pair in the sequence where the second element in the pair satisfies a particular condition and then returning the corresponding first element or a default (i.e. 10) if no satisfactory pair was found.

That code is pretty ordinary. First of all, I would like to improve on the two calls to Seq.tryFind.

let f s =
  let tf g = Seq.tryFind (fun (_,x) -> g x) s
  let o = tf (fun x -> x < 0)
  if Option.isSome o then
    Option.get o |> fst 
  else
    let p = tf ((=) 0)
    if Option.isSome p then Option.get p |> fst
    else 10

Now if the Scala Option orElse and getOrElse functions were available as well as the Haskell backquotes infix syntax then we could really make a difference.

orElse : 'T option -> 'T option -> 'T option
let orElse o p = if Option.isSome o then o else p

getOrElse : 'T option -> 'T -> 'T
let getOrElse o d = match o with | Some x -> x | _ -> d

// N.B. this is not valid F#

let f s =
  let tf g = Seq.tryFind (fun (_,x) -> g x) s
  tf (fun x -> x < 0) `orElse` tf ((=) 0) |> Option.map fst `getOrElse` 10

Neither orElse or getOrElse exist in the F# Option module. However the Core function defaultArg is essentially getOrElse.

Unfortunately F# doesn't have a way to use a (non-operator) function in infix form, like the backquotes in Haskell. However, we can define operators, which are essentially just infix functions.

let (|?) = orElse

let (|?|) = defaultArg

Now we can write a valid F# version.

let f s =
  let tf g = Seq.tryFind (fun (_,x) -> g x) s
  tf (fun x -> x < 0) |? tf ((=) 0) |> Option.map fst |?| 10

Laziness and Composability
Notice that in the original example, the second tryFind is only executed if the first one is unsuccessful because the then expression of an if statement is lazy, i.e. it is only evaluated if the condition is false.

However functions in F# are strict by default i.e. their arguments are evaluated prior to application. Consequently both tryFinds are evaluated, irrespective of their values, as they are arguments to orElse.

So here is an example where laziness is required to achieve reasonable composability. Implementing this gives the final code.

let orElse o (p:'a option Lazy) = if Option.isSome o then o else p.Force()

let (|?) = orElse

let (|?|) = defaultArg

let f s =
  let tf g = Seq.tryFind (fun (_,x) -> g x) s
  tf (fun x -> x < 0) |? lazy (tf ((=) 0)) |> Option.map fst |?| 10

University lectures

I enjoy learning, especially in the context of improving whatever it is I am doing. I do not enjoy the university lecturing system, even when the content is relevant and the lecturer is good. In contrast to interactive learning in the context of relevant activities, it is very inefficient.

In the industrial model of student mass production, the teacher is the broadcaster. A broadcast is by definition the transmission of information from transmitter to receiver in a one-way, linear fashion. The teacher is the transmitter and student is a receptor in the learning process. The formula goes like this: "I'm a professor and I have knowledge. You're a student you're an empty vassal and you don't. Get ready, here it comes. Your goal is to take this data into your short-term memory and through practice and repetition build deeper cognitive structures so you can recall it to me when I test you."

The definition of a lecture has become the process in which the notes of the teacher go to the notes of the student without going through the brains of either.

Provocative but unfortunately sometimes true. In The Impending Demise Of The University Don Tapscott goes on to write about the generational and technological forces that are impacting the traditional university model of learning.

Carl Wieman, recipient of the Nobel Prize in Physics in 2001 writes in Why Not Try A Scientific Approach To Science Education? about his experiences in educating.

I have conducted an extensive research program in atomic physics over many years that has involved many graduate students, on whose professional development I have spent a lot of time and thought. And over the years I became aware of a consistent pattern: New graduate students would come to work in my laboratory after 17 years of extraordinary success in classes, but when they were given research projects to work on, they were clueless about how to proceed. Or worse — often it seemed that they didn’t even really understand what physics was.

But then an amazing thing happened: After just a few years of working in my research lab, interacting with me and the other students, they were transformed. I’d suddenly realize they were now expert physicists, genuine colleagues. If this had happened only once or twice it would have just seemed an oddity, but I realized it was a consistent pattern. So I decided to figure it out.

He goes on to talk about measuring student learning, research on learning, the role of technology and various approaches tried. Towards the end of his multi-part series he offers the following summary.

Our society faces both a demand for improved science education and exciting opportunities for meeting those demands. Taking a more scholarly approach to education—that is, utilizing research on how the brain learns, carrying out careful research on what students are learning, and adjusting our instructional practices accordingly—has great promise.

Research clearly shows the failures of traditional methods and the superiority of some new approaches for most students. However, it remains a challenge to insert into every college and university classroom these pedagogical approaches and a mindset that teaching should be pursued with the same rigorous standards of scholarship as scientific research.

Confused twitter

Suddenly I have no followers on twitter. I wonder what the probability of all my followers stopping in the last hour is?



Also I am only following about 16 people at the moment, but you'll notice about 36 following icons. This is the third time in the last three days that twitter has magically added about 20 followers, which I have deleted again and again. Is this a feature?

Why did the Government give Gerry my money?

“I’ve been in business since 1961 and I’ve never seen this sort of sales decline,” Harvey said. “I don’t know what we would do if we didn’t have Australia growing to make up for it.”

From Harvey Norman to ‘Persist’ in Ireland Amid Mounting Losses

"It is worth noting that the sector least affected (i.e. with little change in hours worked) is retail, no doubt buoyed by the Government cash packages," the report notes.

From 'Armageddon' averted but recession still likely: NAB

If I recall correctly, the first Government 'stimulus' payment was made just before Christmas, in time for the peak retail period and before the effects of the global financial situation were really being felt by the general population.

Firefox 3.5 password manager changes

I use the Secure Login Firefox add-on and store the password files in Git. Upgrading to Firefox 3.5 changes those files.

• signons3.txt is replaced by signons.sqlite


• key3.db is still required as before

Unfortunately signons.sqlite is a binary file and so doesn't play as nicely with Git.

Flickr scare

Just signed in to flickr and realised my pro account had expired and the free usage limits applied, so some of my old photos were missing!

I quickly paid for the pro account and they were back again.

I did not receive an email notifying me of pending expiry at my primary email account (which I have registered with flickr) and I don't check the yahoo mail associated with my flickr account.

Was quite worried there for a few minutes. Flickr has been pretty good, but a bit disappointed about this.

The Chinese Equity Bubble: Ready to Burst

Amid the current financial crisis, there has been one equity index beating all others: the Shanghai Composite. Our analysis of this main Chinese equity index shows clear signatures of a bubble build up and we go on to predict its most likely crash date: July 17-27, 2009 (20%/80% quantile confidence interval).

From this paper.

Only a few days to go before that time period is up! :-)

Ken Robinson says schools kill creativity

A humorous, but poignant view on how our education system devalues creativity in the interests of industrialism to the detriment of people in general.



I disagree with the implication that mathematics is not creative. Paul Lockhart in A Mathematician’s Lament writes about the education system destroying the curiosity and creativity in maths.

I agree wholeheartedly that the education system values certain qualities in people over others and that can deeply (negatively) impact the self worth of young children on the wrong side of that.

Workingmouse, a short history

I was one of the founders of Workingmouse and started as the CTO. Over time the other founders moved on, one to property valuation, another to .net development/project management and the last became the CIO of one Australia's largest retail companies. I took over as managing director, but stayed involved in development and project management until the end.

Workingmouse' history can be considered as consisting of three broad periods, that don't necessarily have distinct boundaries, but serve to characterise some important aspects of the organisation at the time.

1. Central Control
Workingmouse was founded by four software professionals in January 2000 and incorporated the following month. The company grew rapidly to about 19 people, servicing one major client. The Tech wreck of 2000/01 greatly impacted us and so we went through the unpleasant task of letting most of the staff go and then living off our own personal savings/credit to be able to pay the few remaining people.

In those early days we operated in a fairly traditional manner for the time - waterfall style process, multi-tiered Java applications with specialists in each particular technical role, e.g. Java programmer, DB programmer, HTML/JavaScript programmer, system administrator. I was probably the only person technically across all roles.

We grappled with the usual issues of such a situation. How do we do fixed price, fixed scope work and

1. make a profit


2. have a happy customer


3. have happy staff


4. actually do a good job technically

I eventually concluded that it was very difficult (if not impossible in our context) to achieve all four.

2. People/Agile
Initially one of the other founders was highly focused on quality customer service, while I was driven to balance factors such as technical quality and work/life balance (as I had small children with a medical condition at the time). This dynamic, anchored by a developing mutual respect for one another drove radical changes in Workingmouse. Many things we did bucked the local trends of the time and I discovered just how dysfunctional the international software industry is.

The key changes that started brewing in 2001 were a shift from primarily procedural coding to OO domain modeling, waterfall to agile and an emphasis on hiring and empowering talented, passionate, innovative developers. While I describe these individually, they were very much intertwined chronologically.

N.B. this is intended to convey a historical perspective and I don't necessarily hold these opinions or discuss issues in these terms now.

I was driven to optimise our development speed by trying to work smarter rather than longer. When I discovered the idea of modeling the client's problem domain in an OO style, it appealed to me for several reasons.

• The prevailing multi-tiered architecture ideas of the time involved transforming essentially the same data structure between the implementations in each tier. This seemed like a gross waste of time and energy when a lightweight object model could act as a common data structure (and behaviour), utilised across the entire application.


• The object model could predominantly contain and enforce the various 'business rules' and 'validation rules', making them easier to locate and reducing duplication across the application.


• Objects could provide a more natural mapping between the code and the client's description of their world, thus making it easier for the developer to do the mental translations between them. The benefits being efficiency and less complexity and therefore hopefully less errors.


• Application errors could be reduced by composing small robust objects that ensure their interface doesn't provide a way to manipulate them into an invalid state.

Early on we did all our work with simple servlets and straight JDBC in the Resin servlet engine. I thought we had better catch up with everyone else and I investigated EJB around version 2.0. It didn't take too long to figure out that O/R mapping was what we needed, but CMP/BMP was terrible. Tried out Castor for a while, but it had issues and moved on to Hibernate.

The only Application Server that was remotely usable in development was JBoss. At the time I just couldn't believe how vendors could sell such rubbish and customers would actually spend tens of thousands of dollars buying it. Not only that, but they would then incur massive programmer costs trying to work with it. Running WebLogic in development was akin to applying a giant handbrake. We stuck with Resin until Tomcat eventually stablised enough to use it in production.

For some time both clients and Workingmouse had been facing the high cost of dealing with content changes in web sites/applications. We decided to solve this and build a Content Management System. This was our first (and only) product and we sold it as a service. While it was good, I believe it suffered from two business problems. Firstly the market wasn't ready and secondly we targeted the wrong segment (SME when we should have chosen enterprise). Consequently a decision was eventually made to retire it and focus our energies on services, which was profitable.

At some point I investigated the Spring framework and we originally adopted it to discard our own proprietary configuration libraries. It was only later on, when we moved to TDD, that we valued it as a Dependency Injection container. Back when we started with it though, heavyweight EJB and an App Server was "Best Practice". As far as I know we were the first doing Hibernate/Spring/POJO development in Brisbane and possibly in Australia.

Extreme Programming (XP) is the first agile methodology I explored deeply. It looked attractive as it attempted to address the issues we were experiencing.

• Clients don't really know what they want and programmers are terrible at estimating. Short cycle, iterative development with feedback attempts to address this.


• Clients change their mind regularly. XP attempts to positively support that rather than fight it.


• The genuine opportunity to do a good job. Traditionally, everyone pays lip service to quality, but when push comes to shove, other drivers usually take precedence.


• Emphasis on sustainability, of which maintaining a healthy work/life balance is important.

Like many groups, we struggled with getting stakeholder involvement and enterprise structures. Some experiences are documented in this presentation. We explored TDD and pair programming and questioned the costs and benefits in relation to the capabilities of our people.

The name 'Agile' is a misnomer though. I think of sustainable business agility, but most developers think of freedom. Freedom from being held to estimates they couldn't realistically make in the first place or freedom to do a quality job (from their perspective) or freedom to work in a more social/group manner. Technical people try to spin business benefits on top of all this, often with a long-term outlook. However, because of their essentially self motivated position, they don't credibly understand the short vs long term business issues. Consequently hiring existing Agile professionals was sometimes challenging and naturally I don't fit into the orthodox Agile community particularly well.

From around 2003 when we started hiring again, we focused critically and selectively on finding the right people. We looked for technical talent, passion for improvement and honesty with the hope that we could provide an environment for such people to thrive. Sometimes we made mistakes and commercial realities did intrude, but other times I like to think we really got things right. In the Mythical Man Month, Fred Brooks writes of the massive difference in abilities between programmers. This is consistent with my observations of many people employed as programmers, both staff and clients. The differences in productivity and quality are surprisingly large.

When Ruby started emerging as a fashion, we like many Java developers that used TDD, wondered whether Ruby + TDD would be better. It was easy to dismiss the claim of 10x productivity gains, but while skeptical we undertook a few Rails projects. One of those projects was quite long running and exposed issues with reasoning about larger Ruby applications. Our interest in functional programming was emerging throughout this time and it easily eclipsed our interest in Ruby.

3. Functional
In 2006 I employed Tony Morris, who had previously worked for IBM on Tivoli and their JDK. Tony is one of the most intelligent people I know and he seriously challenged our ideas about software development. We thought we were critical and open minded but he took that to a whole new level and many passionate debates ensued. After some investigation into functional languages Tony and I began learning Haskell.

For some people this was a difficult time within the organisation. I was busy unlearning much of what I had built up, while those involved in ongoing projects didn't have the time to be involved in such a journey. I have always tried to be honest with staff and clients and so because of my fundamentally changing ideas I could no longer participate as easily in the general technical discussions. This situation brought about uncertainty as to the ongoing direction of the company.

In 2007 I decided that for all Haskell's elegance it was not going to be a viable option for enterprise software development services in Australia. Therefore we conceded on Scala as a compromise. It is far more expressive than Java, but still executes on the JVM and interoperates with existing Java code, making it far more commercially palatable. We completed our first Scala project in late 2007.

2008 saw Workingmouse in a position to confidently offer general Scala development and training services. Several more of our (highly capable) people had undertook learning Haskell and Scala. It was interesting to observe how difficult the transition can really be. It is relatively easy to learn the syntax of a new language, but much more difficult to fundamentally change the core abstractions one thinks and expresses ideas with.

After breaking the shackles of Java it is painful to go back to it. Workingmouse needed to find some Scala projects, but the global economic situation was deteriorating and so the clock was ticking. In general, technical management is very conservative here in Australia (that may deserve a blog post of its own) and so only one more Scala project was forthcoming. This was not sustainable and with financial markets crashing it was time for this services business to call it a day.

QOTD

I can honestly say if someone had shown me the Programming in Scala book by by Martin Odersky, Lex Spoon & Bill Venners back in 2003 I'd probably have never created Groovy.

- James Strachan

From Scala as the long term replacement for java/javac?

QOTD

The value of a college education is not the learning of many facts but the training of the mind to think.

- Albert Einstein

From Einstein, by Walter Isaacson, p299.

It is a shame we haven't got this sorted out yet.

Workingmouse R.I.P.

After being part of Workingmouse for the last nine years, it is time for a change. The general economic situation, the emotional cost of managing staff and clients, company administration and the ongoing technical pursuit of better software development can make life a little busy and intense. So apart from one or two administrative things, Workingmouse is now completely dormant and only time will tell if it starts up again.

I would like to thank all the people that made it such an enjoyable and interesting place to work. If you find somewhere else like it, please let me know. :-)

Brad Clow
Managing Director

QOTD

Blind respect for authority is the greatest enemy of truth.

- Albert Einstein

From Einstein, by Walter Isaacson, p67.

The Science of Computing and the Engineering of Software

In his keynote at QCon London 2009, Tony Hoare attempts to characterise science and engineering as two ends of a continuum that describes the roles of software people (industry and academia) and their relationships with one another.

The comments about engineers general dislike of mathematics are consistent with my observations. As someone who is currently studying maths, I wonder how much of that is due to the way it is taught at University?

His humble and pleasant responses to the questions at the end is wonderful. I observe a genuine search for truth and clarity with disciplined thought, that is often lacking in much software discussion.

Video and slides here.

QOTD

There are two today.

Indeed the early airplanes didn't come with a specification. You just jumped into them and switched it on and hoped for the best. Much as we do with software today.

The second (in response to the relationship between TDD and specifications).

A specification is just nothing but a generalised test.

Both by Tony Hoare in his keynote The Science of Computing and the Engineering of Software at QCon London 2009.

Scala LinkedIn Tech Talk

Speakers: Martin Odersky (EPFL), Nick Kallen (Twitter), David Pollak (Lift)

Martin gives a good overview of who is using Scala, where it came from and where it is going in the first 16 minutes.

Throughout the video, Nick provides the most plausible comments I have heard to date regarding Twitter's motivation/experiences in moving from Ruby to Scala.

F# Interactive on Mac OS X

Installing F# on Mac OS X is easy (with MacPorts):

$ sudo port install fsharp

I have version 1.9.4.19 installed. Microsoft released a newer version several weeks ago, however it doesn't seem to have made it into MacPorts yet. Then to start F# Interactive:

$ fsi --readline --no-gui

There is basic history with the up/down arrows, but the usual emacs key bindings are not available, so entering and editing text is very slow and painful. I tried to fix this with rlwrap.

$ sudo port install rlwrap

As fsi is already using readline, rlwrap needs the -a option, See the man page for details, but on the mac you are required to provide a string (representing a password prompt the wrapped application might show) argument for this option. It is completely unnecessary for me at the moment, so I just picked something obvious as a placeholder.

$ rlwrap -aPassword: fsi --readline --no-gui

F# Interactive starts up with the usual splash text, but there seems to be some control characters automatically input and the console clears to something like this:

- 0;3R0;3R;3R;3R;3R;3R;3R

The emacs key bindings work, so this text can be deleted with C-a C-k and terminating the now empty line as per normal in fsi with ;; brings up the usual fsi prompt.

Unfortunately, if you type in an invalid expression eg. x = 3;;, the cursor is placed in the middle of the error message. When pressing up arrow to go back through the history, sometimes the prompt is replaced by partial text from a viewed history line.

So this is a pretty dodgy solution. If anyone knows how to get it to work properly, please leave a comment.

BCA v Singh The Story So Far 3 June 2009

I didn't realise Simon Singh was being sued for libel by the British Chiropractic Association until I read Tony's recent post.

Simon's account of the situation is here. It would be nice to read his original article that the BCA is taking action over, if someone can find it.

Commercial Uses: Going functional on exotic trades

A peak into how derivatives trading works at Barclay's and their supporting Haskell application.

QOTD

As we performed the work outside of a traditional it team, management was willing to take a risk on something that promised a good fit and rapid development process.

Abstract

The Functional Payout Framework, fpf, is a Haskell application that uses an embedded domain specific functional language to represent and process exotic financial derivatives. Whereas scripting languages for pricing exotic derivatives are common in banking, fpf uses multiple interpretations to not only price such trades, but also to analyse the scripts to provide lifecycle support and more. This paper discusses fpf in relation to the wider trading workflow, and our experiences in using a functional language in such a system as both an implementation language and a domain-specific language.

The paper is here.

Backup Google Docs

I want a command line tool to copy all my data from Google Docs to my local machine. So far I have found 3 ways to download data from Google Docs, but none are satisfactory.

Google Docs: Download Greasemonkey script runs in Firefox.

GDataCopier looked like it would do the trick, but the current version (1.02) only seems to work with documents and spreadsheets. The feature list for the forthcoming version adds support for presentations, but there is no mention of PDFs.

GDocBackup doesn't work with PDFs and doesn't seem to have a command line option. I already had mono installed on my MacBook, so installed GDocBackup to try it out. First I got an cxception that boiled down to:

 System.DllNotFoundException: gdiplus.dll

After sifting through these: DllNotFoundException, OpenSim Troubleshooting, I added the following line to /opt/local/etc/mono/config to get it working:

<dllmap dll="gdiplus.dll" target="/opt/local/lib/libgdiplus.dylib" os="!windows"/>

Future of Computing

Some quotes from The Future of Computing: Logic or Biology, by Leslie Lamport.

• Computers interact with users through metaphors. Metaphors are not based on logic, but they are not illogical. Metaphor and logic are not opposites. They are complementary. A good program must use good metaphors, and it must behave logically. The metaphors must be applied consistently—and that means logically.

• Floyd and Hoare pointed the way by showing how mathematics could be applied to tiny programs. We need to learn how to extend what they did to the real world of large programs.
  
  Extending what they did does not mean trying to mathematically prove the correctness of million-line programs. It means learning how to apply the idea of a program as a mathematical object to the task of building large programs that we can understand, and that do what we intend them to do.

• In addition to the task of learning how to apply mathematics to large systems, we also face the task of teaching programmers and designers to think logically. They must learn how to think about programs as mathematical objects. They must learn to think logically. We will not have understandable programs as long as our universities produce generation after generation of people who, like my former colleagues, cannot understand that programs are different from automobiles.

Patent Lens

Around the time I was experiencing the patent system firsthand, I coincidentally had lunch with Richard Jefferson of Cambia. Cambia is a not-for-profit organisation that is moving some of their operations from Canberra up to Brisbane to collaborate with QUT.

Richard was interviewed on ABC Catalyst where he talks about:

• applying open source s/w development ideas to life sciences


• more efficient social change through scientific collaboration and transparency


• the BIOS initiative that provides tools for genetic scientists (at no financial cost) with the aim to help local people solve their own problems rather than just relying on aid

One of the things Cambia does is the Patent Lens, a free patent search engine. From their site:

We created the Patent Lens to create transparency in the patent system, and to serve the public worldwide as a platform resource to explore, understand and improve its impact on society.

The effects of the patent system as it works now may not match the original intent to benefit society.

The patent system was created to advance societal benefit by encouraging public disclosure of inventions and clear definition of each invention, in exchange for a strictly limited monopoly. The intent is that the invention may be used wherever and whenever the patent monopoly is not in force.

The trouble is that unless information is readily available about whose patents are in force over what technology where, the system doesn't work well.

People may unwittingly infringe patents they don't know about, avoid areas of innovation in which they are entitled to be creative, or make poor investments based on incomplete information about which rights are granted and who holds these rights.

The Patent Lens informatics tools can assist the user to determine the boundaries of intellectual property constraints on deliverable innovations, and usable building blocks for future innovations.

Stanford Lawyer: interview with Charles T. Munger

I would argue that a majority of the horrors we face would not have happened if the accounting profession developed and enforced better accounting. They are way too liberal in providing the kind of accounting the financial promoters want. They've sold out, and they do not even realize that they've sold out.

Charles T. Munger, vice chairman of Berkshire Hathaway (Warren Buffet's company) in
Q&A: Legal Matters with Charles T. Munger

Assuming Charles' statement is correct, I don't think accounting is the only profession that has sold out to mediocrity, driven by commercial benefits to a minority, but to the overall detriment of society.

5 Software guys walk into a bar ...

Well it was actually a restaurant, not a bar. Anyway, while we were there, Dan North asked me what I thought about Clojure, to which I replied that I preferred strong, statically typed languages. This sparked a huge debate between Dan, Joshua Bloch, Dave Thomas, Bas Vodde and I.

I managed to describe a little of Kristian's Ruby example, which received some agreement. However the best part, was probably Dave and Josh almost yelling at each other trying to get their point across. Admittedly the restaurant was a bit noisy, although I think we were making most of it. :-)

I had a great time, thanks for dinner guys.

JAOO Brisbane 2009

The JAOO 2009 Conference in Brisbane finished on Tuesday.

Day 1
I missed most of the days program, although I did get to hear Clemens Szyperski talk about MEF and Oslo at QUT in the afternoon.

Jonas S Karlsson from Google talked about Consistency, Storage, and Reliability in the Cloud. He was involved in the Google Megastore project which provides a scalable (in Google terms) storage layer with SQL style schema and secondary indices and consistency. I believe he made a comment about the importance of the dollar cost per user (of a hosted app) to be a single digit, especially when the app is available free of charge.

Mike Cannon-Brookes finished the day off with lessons learned at Atlassian.

Caught up with various people at drinks afterwards and went out to dinner with some of the presenters at SouthBank.

Day 2
Joshua Block gave the keynote on various technicalities in Java. It is not often that someone mentions covariance and contravariance in a Java talk! Seems to come up more in Scala discussions and then people are scared off by the terminology. :-) While probably not the intent, I think the talk was great in that it clearly demonstrated some of the complexity of Java.

Douglous Crockford, Writing Large Applications in JavaScript. Interesting talk, although I think it was a mistake for him to use Java as an example of strong, static typing in his argument for the benefits of loose typing in JavaScript.

Mahout by Jeff Eastman. This was given on a few hours notice to fill in for someone who was sick. I have done stuff with Hadoop before and am developing an interest in machine learning, so I found this talk interesting. There was someone in the audience who was a little confused though, as he thought Jeff meant java.util.Vector whenever he used the term vector. :-)

Unfortunately Joe Duffy was sick, so his talk on parallel programming was canceled.

The last talk I went to was Joel Pobar on F#. It was great to hear from a local who knows some stuff about functional programming (FP). He was suffering from a cold, but sill powered through some interesting algorithms. I don't think he helped the mainstream perception that FP is really technical (given that many are scared by this) when he mentioned vector spaces. :-) We talked to Joel afterwards about the Brisbane FP group starting up and he was really excited, so it will be fantastic to have him involved.

QOTD

Regarding JavaScript:

Given the process we use to create web standards we deserve something much worse.

Douglas Crockford at JAOO Brisbane.

GMail down

GMail disappeared a few minutes ago, still down.

EDIT: Back up again within 10 minutes.

Patents can hinder innovation

Recently I was involved in a startup that after commencing discovered patents that they may be infringing. Basically, the owner of the patents had tried to setup a business and failed. Now however, the owner is in a position to profit (or recoup investment losses) if they could sell the patents, or alternatively bring a claim against the startup (or anyone else) once it launched.

My understanding is that the intention of patents is to foster innovation by granting a limited monopoly so as to encourage the investment required. However in this case, the patent holder has failed to bring their "innovative" idea to market and is stifling others in doing so by the risk of an infringement claim.

Similar issues are covered in Grove Says Patent System May Have Same Flaws as Derivativesand Former Chief Executive of Intel Believes Patent System Has to Be Altered.

Google Code supporting Mercurial

Google has announced support for Mercurial on Google Code. Currently it is a preview release for a limited set of users.

There is a Google analysis of Git and Mercurial here.

UQ switches to MS Exchange Labs

The University of Queensland is switching over to Microsoft Exchange Labs for student email accounts. In the past I have redirected all my university email to my personal GMail account and I wished to continue that.

This turned out to be a ridiculously complicated process, that I would have never figured out without the instructions. Furthermore it can only be done in IE 7! Obviously the IT staff haven't walked around with their eyes open, otherwise they might have noticed how many students have MacBooks.

From my brief exposure to this service, it seems like a really bad idea to try and clone a desktop application (Outlook) in a web browser. Having to double-click is completely inconsistent with the normal browser paradigm.

QOTD

Java is the Brier Rabbit of IT. Once touched you can't let go. Its simplistic enough to be inadequate in almost every situation.

From The Book Of JOSH - Scala In The Enterprise

MacBook water resurrection

Over Easter, my (adorable) 4 year old daughter was watching a DVD on my MacBook and she spilt a cup of water on it. When I got to it, the MacBook had water on the keyboard and had stopped working.

After wiping off the liquid on the outside, I pulled out all the parts that are fairly easy to access - battery, RAM and hard drive. The hard drive casing was wet. One RAM stick had a few small drops on it and both had a white powder on the contacts. I wonder if this was a residue from rapidly evaporating water. I propped the MacBook upside down, with the parts still out, hoping it would dry overnight. I didn't hold much hope.

The next morning I put it back together and tried to power it on, but no go.

We traveled home Monday and that night I plugged it in and it actually booted up. There was a wet patch in the bottom of the screen, but otherwise it was working fine. I couldn't believe it.

I found this post, so the next day I tried it. Two terminals executing yes > /dev/null to max out the CPU and the screen open at about 45 degrees, to try and catch the heat. About 4 hours later the wet patch was gone.

I am writing this post on the MacBook. Amazing!

My Manhattan Project - how I helped build the bomb that blew up Wall Street.

For those of us that consider job satisfaction in more ways than how well we do our job or how well we are paid, but also how does our work impact our world? Does it simply enrich some nameless shareholders, or does it do more?

Here is the story of Michael Osinski, a programmer that wrote software that turned mortgages into bonds. Somewhat relevant in the current "global economic crisis".

The original article, spread across multiple pages is here.

Netbooks

A few years ago I used to use an Apple 12" PowerBook and it is still my favourite laptop. I now have a 13.3" MacBook, that seems too large to be convenient when on the move.

When working at a semi-permanent desk I prefer to use a desktop, rather than a laptop anyway. So I need a machine that is just powerful enough for my day-to-day, non software development activities such as email, calendar, tasks, documents, music, skype and reading the web. Most of these are accessed through a web browser anyway. It also needs to be small enough to fit in restricted spaces and not be a burden when walking/cycling.

Given these requirements a Netbook seems a logical consideration. I wandered into a retail shop this morning and the keyboards on all the smaller ones with 8.9" screens were just too difficult to use for my fingers. The models with 10" screens seemed ok though.

All the following are only offered with Windows (in Australia): Dell Mini, BenQ Joybook Lite U101, Fujitsu M1010, Acer Aspire One 10.1, HP Mini 2140, MSI Wind series, ASUS Eee PC 10" series, which I don't find suitable for stability and usability reasons. While the Eee PCs apparently have a Xandros option, I could only find them being sold with Windows. It is also interesting that ASUS gives better battery life figures for Windows than Linux.

The remaining options seem to be the Kogan Agora Netbooks which come installed with gOS, based on Ubuntu. Or alternatively, buy one of the Windows ones that can be reinstalled with linux. The BenQ Joybook seems to be the cheapest of the bunch, coming in at around $600. The Kogan Agora Pro has more RAM (2GB in total) and is priced at $539. While previews indicate good build quality, this is Kogan's first foray into computers, so there is the usual version 1.0 risk.

The future looks interesting - ASUS has announced the Eee PC T91 and T101H, both with touch screens.

The coolest though, would have to be the Touch Book by Always Innovating. The screen and keyboard detach so you can use the screen on its own as a tablet device. I suspect that would be more comfortable when browsing/reading away from a desk. It is also magnetised so you can stick it on the fridge. I would be much happier with my wife accessing her recipes that way, rather than having the MacBook on the kitchen bench surrounded by ingredients. Finally, it has 10-15 hours battery life. Unfortunately it won't be available for several months yet.



Some useful sites for research were cnet.

Questioning the Windows standard

I don't really follow this sort of thing, but I noticed a story on the French police switching from Windows to Ubuntu. Even if the numbers are wrong, it is nice to see a large organisation that has actually not followed the defacto standard of Windows, which is what I see in the organisations around me.

Tony's letter to the Medical Board of Queensland

Edit 19/10/2010: I have removed the name of a doctor after receiving a letter from his lawyers. Details here.

We received a copy of this yesterday evening. Please note Tony is a friend of mine and this is genuine.

To The Medical Board of Queensland,

My complaint is rather lengthy. It relates to an injury and subsequent treatment of that injury by various doctors and surgeons. I am not primarily complaining about any particular individual of the medical institution, rather, that I still have not been diagnosed and treated for my condition. I am desperately seeking diagnosis and a treatment. Some doctors may believe they have diagnosed my condition, but they have been wrong and this has been proven surgically. The best diagnosis available today is my own humble conjecture, which is terrifyingly inadequate.

On 28 July 2007, I suffered a severe inversion sprain to my right ankle. I'd suffered a few previous sprains, but this one was much worse than others. I was kicking a football rather hard when I landed incorrectly. I was not weight-bearing for about 5-6 days. During that time I was a very fit athlete playing A1-grade tournament squash and was very active with many other hobbies.

Over the following months after the injury, I tried returning to sport and I could feel I had something wrong in my ankle. I hoped it would resolve, but it didn't. I could not put my joint into dorsiflexion due to a mechanical impingement and this resulted in muscle atrophy. Subsequently, I became very ill. Nevertheless, I pushed on with my sporting endeavours expecting my body to overcome the problem like it had many others in the past.

Eventually I conceded in January 2008 and sought help from Dr. X. We tried various treatments including a cortico-steroid injection and obtained MRI radiographs. Over time, this condition worsened to the point where I was forced to discontinue sport in May 2008. I have not been active since. I sought help from an orthopaedic surgeon, Dr. Greg Sterling, who prescribed a Broestrom Repair and medial arthroscopy since my complaint was mostly anteromedial.

At the time, I was very medically-illiterate and put much faith in medical doctors. I assumed a positive outcome from surgery, simply because I'd had surgery in the past for various conditions and I was always better afterward. The procedure was performed on 15 September 2008 and the medial arthroscopy revealed deltoid ligament damage which was also repaired along with the ATFL and CFL.

I wore a cast for 2 weeks and an orthodic boot thereafter. Although I was in pretty intense pain, I'd attributed this to the recent surgery and thought it would resolve. It didn't. In October 2008, I knew I was in big trouble – I had similar symptoms to pre-operative but they were now much worse. I immediately sought help from Dr. Greg Sterling who requested another MRI but in November 2008, could see nothing wrong. He said “see you in January 2009”. The prospect of waiting so long in agony was traumatising.

Around the same time, I'd sought help from Dr. X who claimed I was exaggerating my symptoms and attributing too much attention to my condition. This blatant oversight added further to my trauma – and he'd almost convinced a member of my family of these falsehoods. I was suffering psychologically and I sought immediate help from Prince Charles Mental Health Unit.

I was discharged as an outpatient but I knew I was not mentally ill – I was in incredible pain and agony from a misdiagnosis by an orthopaedic surgeon. I set out to conquer the problem myself – scared and ill-educated on the subject matter – and was faced by the foreign medical protocols and language.

As my stress levels grew, I was admitted to the Prince Charles Mental Health Unit by Dr. John Reinders under a de facto ITO. I was also forced to take anti-psychotic medication. This is because of some of the desperate language I was using, for example, “do I have to operate on myself?” and Dr. Reinders felt I may be a harm to myself.

I'd never been so low in my life. When taking anti-psychotic medication, you are very much unaware of your surroundings. It was only during a lull in the effect of the medication, that I decided I needed to get out of the hospital and do what I can for my ankle – I was convinced the doctors had erred and that this was a huge mistake. I requested a Psychiatric evaluation and was declared “in severe distress, but mentally healthy” and I was discharged.

I set about understanding ankle anatomy, conditions of the ankle and general medical protocols. I quickly learned that I had at least soft tissue impingement. Indeed, I had tissue trapped in the joint that was under permanent pressure due to the recent surgery – even when not weight bearing. This is as painful as you might imagine it to be and a little more given multiple pathologies.

I sought help from Dr. Andrew Wines (Foot & Ankle Orthopaedic Surgeon) in Sydney who prescribed an arthroscopic debridement. This was performed under GA on 11 December 2008. To quote his remark, “you had a chunk of tissue about the size of my finger in there”. I was immediately weight bearing post-operative and the local aneasthetic provided some relief. As a result of this anaesthesis, I was under the false impression that my troubles were over. They weren't.

After a few days I knew I still had a severe and painful problem though I no longer had the problem of tissue impingement. It felt like I had bone impingement on dorsiflexion and I sought answers from medical literature. I eventually stumbled on Anteromedial Osseous Impingement Syndrome for which MRI radiography is inconclusive for diagnosis. I used my January 2009 appointment with Dr. Sterling to ask for a request form for a CAT radiograph and weight-bearing Xray. Dr. Sterling also made an appointment with Dr. Michael Lutz to determine if he could unravel this mysterious problem of mine.

Upon obtaining these radiographs, I saw immediately that I had a bone spur in the location of my pain. I sought assistance from Dr. Andrew Wines (again in Sydney) who agreed with me to some extent but wanted a second opinion in order to ensure he was not suffering a bias. I applaud this decision. I used my upcoming appointment with Dr. Lutz to achieve this second opinion. Dr. Lutz agreed that an open incision to excise osteophytes on my tibia may be appropriate and I was informed of the risks .

Dr. Wines performed an arthrotomic tibial ostectomy on 04 March 2009 under GA at Royal North Shore Hospital. I flew home the next day – I was weight bearing without assistance. Again, the anaesthetic provided a false belief that my problems were over.

Unfortunately, I still have the same problem I started with – I cannot put my ankle into dorsiflexion. Many doctors might attribute this to the recent surgery, but I know I am experiencing precisely the same symptoms that I have done for the last 18 months. Although the localised and extreme pain caused by a bone spur has been resolved, I still have bone impingement on dorsiflexion that is not localised. I also know that this is due to a very definite mechanical limitation, since I have had a Physiotherapist in the past attempt to push my joint past this point with intense force to no avail.

As a result of this inability for dorsiflexion, my muscles have atrophied up to my thoracic spine. Subsequently, I find breathing difficult and any position uncomfortable except for lying down. This has caused immense stress especially while I have maintained a full-time job (Computing Science Researcher) and I am considering indefinite unpaid sick leave. Unfortunately, a consequence of this is that I will no longer be able to afford regular flights to Sydney for treatment, radiographs and so on, therefore, I must continue the battle under all circumstances and despite exhaustion.

I have long maintained that my symptoms are only observable while I am weight-bearing and in an attempt at dorsiflexion. This means that surgeons who operate cannot check for resolution of these symptoms, MRI and CAT radiographs cannot exhibit them and the only weight-bearing Xray I have was not in dorsiflexion because the Radiographer would not allow it (images only per requesting doctor instructions).

Unfortunately, I still have quite a large battle ahead but I am at a complete loss with respect to how I should go about it and that is why I have written to you, my state medical authority.

Is there a radiographic machine available in Queensland that will exhibit my bone impingement while in dorsiflexion and while weight-bearing? Better still, is there a doctor who is prepared to do the hard work of figuring this problem out? I am more than willing to make many sacrifices to ensure it. These are rhetorical questions – I don't know what the right questions are.

I am desperately seeking a diagnosis and treatment, 19 months after an initial injury and I know that my time is limited with regard to the amount of stress and pain that I can continue to endure. This is simply unsustainable.

Please advise.

Thank you for your time. Confidentiality is not requested and you may share this information with whoever you see fit.

Books and libraries

I have decided to reduce the number of books I buy. I only intend to purchase reference books that I think I will use regularly enough over time. I hope to satisfy all my other book needs via the local public and university libraries.

I arrived at this decision due to the environmental, financial and time costs:

• Acquisition. There are all the environmental costs in production, transportation, wholesaling and retailing.


• Storage. You need bookshelves, space to put the bookshelves and energy to light/heat/cool the space for the bookshelves. Then you can spend time organising, cleaning and tidying the bookshelves. Re-arranging and moving become much larger jobs, the more you have.


• Disposal. All the time and effort in getting rid of them responsibly, e.g. finding someone else who might use them rather than just throwing them in the recylce bin.

These issues apply to many things and the earth only has finite resources. As a global society, especially Western society, we need to dramatically reduce individually acquiring/consuming stuff.

Imaginary numbers

The names imaginary numbers and real numbers seem a little ironic, given that all maths is imaginary. As an example, consider the number 3. It is not a real thing in the sense that I can't go and observe a three, like I can stone or a tree. It represents the abstraction or concept of threeness, which we have imagined and exists in our minds.

One way to think of maths is as the science of patterns. So we have a huge library of patterns, abstractions, relationships and methods of reasoning about them. When we want to talk about the "real" world, we often draw on this library to give us the vocabulary and tools to do so.

I was thinking about this stuff because I am studying linear algebra at the moment and was reading A Visual, Intuitive Guide to Imaginary Numbers and What is Experimental Mathematics?

PDF documents in Google Docs

I use Google's PDF Viewer to view my PDF documents stored in Google Docs quite a lot. Recently I ran into two issues with PDF support:

• The PDF Viewer only shows the first 100 pages. The entire PDF is stored in Google Docs, but you have to download it to view subsequent pages.


• I have installed Google Gears for offline support, however PDF documents are not available offline.

This is alongside the fact that you can only upload a single PDF at a time and there also seems to be a limit of 100 PDFs in your account.

Moving Virtual Box VM to another host

I have Windows XP running in a Virtual Box VM. The iMac it was running on is getting intermittent screen artifacts, so time to switch machines.

I found this, which led me to believe that moving hosts is possible. The VM is named "Win XP" and has one snapshot. I mucked up the transfer the first time. These are the steps I followed the second time, which worked.

1. Moved ~/Library/VirtualBox/HardDisks/Win XP.vdi to the same location on the new host.


2. Copied ~/Library/VirtualBox/Machines/Win XP to the same location on the new host.


3. In the old VirtualBox.xml there was a HardDisk tag with the attribute location="HardDisks/Win XP.vdi" that contained a child HardDisk tag (I had one snapshot). Copied these into the new VirtualBox.xml.


4. Copied the appropriate MachineEntry tag from the old VirtualBox.xml to the new one.

The VM started up on the new host with no problems.

Also stumbled across a site with downloadable Virtual Box images.

This is how Scoodi is intended to work

I get an RSS feed of all items listed on Scoodi within 10km's of my house.

Someone posted a wanted listing for CD cases a few days ago. I had some that had been sitting in the cupboard for a long time, so I replied this morning. Since he lives in the same suburb, he walked around and has just picked them up. Nice and quick and easy.

Not only was he very appreciative, but he was also happy that he has been able to get rid of the stuff he has been cleaning out of his garage on Scoodi.

Haiku

BeOS was yet another innovative technical product that was built too far before its time and has subsequently passed away. I bought a copy in the late 90's and still have the CD lying around at home.

The BeOS legacy lives on, in the form of Mac OS X's file system metadata features and as I have just stumbled across, the open source Haiku project. It would be nice to have an open source, simple, consistent, fast but lightweight desktop operating system option.

Fun with functors

I have been slowly working through some more Haskell with help from Tony.

1. Quickly worked through the previous stuff we did to refresh my memory.


2. Consider a simplified version of the Eq type class.

   class Eq a where 
     (==) :: a -> a -> Bool

   The type variable a is polymorphic over types. Now consider the Functor type class.

   class Functor f where
     fmap :: (a -> b) -> f a -> f b

   Here we have an example of higher-order polymorphism, e.g. f a, f b. The type variables a and b are polymorphic over types as before, but f is polymorphic over type constructors. Note that f has kind * -> * as it is applied to a single type variable in both cases.


3. The Maybe type constructor has kind * -> *. We have effectively already written the Functor implementation, i.e. the mmap function in the previous work.

   class Functor' f where
     (<$>) :: (a -> b) -> f a -> f b
   
   instance Functor' Maybe where
     (<$>) = mmap



4. Observe that -> is a type constructor with kind * -> * -> *. Given two types a and b, a->b is the type of functions mapping elements of type a to elements of type b.
   
   We can't write a Functor instance for the -> type constructor as their kinds are different. However we can partially apply a polymorphic type, lets call it t which gives ((->) t) and has kind * -> *.


5. Exercise: write an instance of Functor for functions:

   instance Functor' ((->) t) where
     (<$>) = error "todo"

   Looks a little nasty when you first see it. Its a bit easier if you write out the type signature with the type constructor applied.

   <$> :: (a -> b) -> ((->) t a) -> ((->) t b)

   It is more obvious in infix form.

   <$> :: (a -> b) -> (t -> a) -> (t -> b)

   Now it is fairly easy to come up with an implementation.


6. Exercise: just for fun, write a Monad instance:

   class (Functor' m) => Monad' m where
     flatMap :: (a -> m b) -> m a -> m b
     pure    :: a -> m a
   
   instance Monad' ((->) t) where
     flatMap = error "todo"
     pure    = error "todo"

   Again writing out the type signatures helps.


7. Note the laws that all Functor instances should satisfy.

   fmap id == id
   fmap (f . g) == fmap f . fmap g

   The combination of the Functor type class and these laws define a particular concept/pattern. Intuitively function composition and the traditional map function seem like very different things, yet they are both instances of this abstraction.


8. Exercise: write an implementation for:

   (<***>) :: (Functor f, Functor g) => (a -> b) -> g (f a) -> g (f b)

   It took me a while, with help, so the answer is at the bottom of the post.

This is all well and good, but what use is it? Here are two examples of using the <***> function.

Example 1

data Connection = Connection

newtype Conn a = Conn {
  conn :: Connection -> a
}

instance Functor Conn where
  k `fmap` Conn z = Conn (k . z)

data ResultSet = ResultSet
data Person = Person

f :: ResultSet -> Person
f = undefined

executeQuery :: String -> Conn ResultSet
executeQuery = undefined

k :: String -> Conn Person
k = f <***> executeQuery

Example 2

g :: Maybe [Bool] -> Maybe [String]
g = (<***>) show

Useful references.

1. A Gentle Introduction to Haskell: 2 Values, Types, and Other Goodies


2. Functional Programming with Overloading and Higher-Order Polymorphism

Answer for 7.

(<***>) :: (Functor f, Functor g) => (a -> b) -> g (f a) -> g (f b)
f <***> g = (f `fmap`) `fmap` g

Or in point free form.

(<***>) = fmap . fmap

[shell-fu]$

shell-fu.org has lots of little command line tips and RSS feeds to keep an eye on new ones.