Drivel about data and musings about metadata.

I’ve been recently toying with the idea of using music as a format of exploratory data analysis. While the use of sound to monitor data isn’t new, its still relatively uncommon. As I occasionally find myself trying to make sense of large data sets finding a way to quickly analyse them, to find the points of interest can be quite tedious. So I thought about ways someone with no music skill could generate sound from data, and produce something relatively melodic, and useful for highlighting patterns and anomalies in the data.

Sing a song of software,

To test this out I put together a little tune, that covers the past 24 years of stock information from Microsoft (Piano), Apple(Clarinet) and Google(Xylophone). The pitch is proportional to the price of the stock with low tones being low prices and high be high. While the volume of each instrument is proportional to the volume of sales over the period, so when you hear a quiet sound that is a low volume day, while loud note is a period of higher trading volume.

There are two versions of the music available:

A shorter 2 minute, up-tempo version using weekly stock prices: OGG, Midi – This one is short and to the point, but some of the nuances, like big daily trade spikes are missed.

A longer, 17 minute, version using daily prices: OGG, Midi – This one is a little monotonous at the start, but you can hear Apple come from a tiny instrument in the background to a larger force much better. It also lets you hear some off Apples big trading days.

Bubbles full of lies;

A few things to listen out for:

• Early on, listen for Microsoft’s speedy accent during the 2000′s tech boom, and an even quicker decline. (About 1:00 in on the quicker version)
• Apple, has for a long time a consistently low trade volume, however occasionally you will hear loud piano strikes starting from the early 2000′s. (About x minutes in.) These are peaks of stock sale, probably around MacWorld and iPod/Phone/Pad announcements.
• After about 2005, you can hear Apple and Google slowly rise in volume and stock price, while Microsoft remains in a consistent range throughout the same period. (After 2:00 in the short version)

4 and 20 years of stocks,

The data that all of this was pulled from was the historical stock prices data sets available on Google Finance. Why 24 years worth – because it fit with the theme of the nursery rhyme I was trying to mimic. Its pretty touch and go as to what data you can download from Google Finance, but to be fair, from my understanding this is an issue with the exchanges rather than with Google.

Audio-lised with Py(thon).

So the nitty gritty on how it works:

Its a python script that loops through a set of files of output data from Google Finance and using midiutil creates a Midi file. Each day (or weeks) datapoint is weighted so the values remain within a specific range for a specific instrument and the volumes are adjusted so that each instrument can be detected. Without either of these it really is quite a mish-mash of sound.

This output Midi file is then run through Timidity++ to create an Ogg/Vorbis file. Converting to Ogg is only necessary for consistency, but both the Midi and the Ogg are available.

Future work and ideas

Well the goal is to be able to use a technique like this to listen to large multi-variate datasets, that have either a time dimension, or a continuous dependent variable (heights, weights, etc…). As long as one dimension has values that are relatively evenly and closely distributed with few overlaps and a wide enough spread it should be possible to ‘graph’ probably any dataset meaningfully as audio.

Chess game by niallkennedy, on Flickr

There was a recent post on the Freakonomics blog that mentions a study from the University of Buffalo that attempts to examine the relative strengths of top ranking chess masters over the years to attempt to answer the question “are chess ratings over inflated, or are we really getting better.”

The results of the study showed that chess ratings have not inflated, and we are in fact getting better at chess.

I am in two minds about the fact, at first glance this seems like a real “well-duh” moment in science, but after a little thought it actually comes across as an extremely interesting study.

It seems obvious that we as a society are advaning. Regardless of your perceptions of the “youth of today” or the “invasion of foreigners” (it always seems that conservatives think that young or alien people are just going to destroy society), it makes perfect sense that we are generally getting better at stuff.

The best example of this the progression of skills in skateboarding. While chess has been around for hundreds of years, skateboarding started in the 1940′s, but only got big in the 70′s. This means people have been around long enough to see vast improvements in the sport. For the longest time most skateboarding tricks were flatland based – that is they were either tricks one did while rolling, carving in pools, or handstands. It wasn’t until the early 80′s that the ollie (or jump) became a phenomenon. Without the ollie, almost no modern skateboarding tricks are possible.

It took 40 years for skateboarders to learn how to jump! Now days, it only takes a solid afternoon of practice. This means that anyone who starts skating today can benefit from the knowledge 20 years of study of the ollie.

The same should hold true for almost any discipline – especially chess. Unlike skateboarders, chess players are very meticulous about recording and researching their hobby. In fact Chess.com has records on games earlier than 1940, and chess books will often have games from the 1800′s. So while it may take a lot of practice to advance in chess, usually much more than an afternoon, there still exists an exhaustive wealth of knowledge for newcomers to tap into if they wish to progress and provides the proverbial shoulders for them to stand upon to improve the field.

The reason that this study is fascinating, is that the above really is anecdote. Sure, we can see that we’ve improved at skateboarding or chess. This study takes the vast amount of data available for study in chess and demonstrates that this shared phenomenon or improvement is a reality. We as a society are advancing (at least at chess and skateboarding), and although there is no definite proof of causation, this provides support to our anecdotes.

This post also appears on Chess.com.

The day is finally here – Virgil c2d is available for Windows. You can download the zip archive from Google Code. In future this will be the place that new versions of the tool will be made available, and I am hoping that as people start using it and bug do get noticed that there will be activity, so be sure to check back often to see if changes are available.

For the time being though, download a copy of the beta, checkout some of the example CSVs and  learn about how the different CSV types look.

If you have issues getting the application to run, check the converter_ui.exe.log log file for any errors and be sure to raise a bug through the issue tracker.  If there are issues getting a file to covert check the structure settings are correct, and check the line that the error dialog indicates may be causing the issue. If you are still unable to get the CSV to convert raise an issue and attach the offending CSV file and I’ll see if the problem can be resolved.

When checking out the example CSVs the filenames give some hints to the structure of the data in them:

• ss: semi-structured
• mono: monolingual
• pd: pre-defined language
• pe: prefix embedded language

For the other files they have type:

• anzsic 2006 – codes and titles.csv — Semi-strucutred, Monoglingual
• anzsic.csv – Semi-strucutred, Monoglingual

The solution is now available below

DDI can be a harsh mistress sometimes, and mistakes can sometimes be made when trying to use it. As a data format it is flexible enough to handle most situations, but this flexibility can sometimes be a shortcoming.

Below is a poorly written chunk of DDI I’ve written, that forms part of a survey instrument. The good news it can be written in a much better way. The challenge is how to rewrite it:

<d:Sequence id="MainSequence">
    <d:ComputationItem id="CompItem1">
        <d:Code>
            <r:Code programmingLanguage="pseudoCode">SET X = X + 1</r:Code>
        <d:Code>
    </d:ComputationItem>
    <d:IfThenElse id="ifblock1">
        <d:IfCondition>
            <d:Code>
                <r:Code programmingLanguage="pseudoCode">X == Y</r:Code>
            <d:Code>
        <d:IfCondition>
        <d:ThenConstructReference>
            <r:ID>A_different_sequence</r:ID>
            <r:ID>MainSequence</r:ID>
        </d:ThenConstructReference>
    </d:IfThenElse>
</d:Sequence>

If you think you can correct this code, email a solution to theodore.therone at gmail.com. At the end of the week (When I wake up this Saturday AEST) I’ll select a solution at random and give away a $15 voucher for 5senses coffee.

If you need clarification on anything in the example code, post it in the comments and I’ll clear it up.

Unfortunately, there were no correct responses, so the voucher will go to the next challenge, but the answer is still available below.

Solution – this was a simple computer science riddle wrapped in a layer of DDI. It was a loop rewritten as a recursive if-branch. Rewritten as a loop it comes out as this:

<d:Loop id="MainLoop">
    <d:LoopWhile >
            <d:Code>
                <r:Code programmingLanguage="pseudoCode">X == Y</r:Code>
            <d:Code>
    <d:LoopWhile>
    <d:StepValue>
        <d:Code>
            <r:Code programmingLanguage="pseudoCode">SET X = X + 1</r:Code>
        <d:Code>
    </d:StepValue>
    <d:ControlConstructReference>
        <r:ID>A_different_sequence</r:ID>
    </d:ControlConstructReference>
</d:Sequence>

A much cleaner solution!

When I’m not writing about writing code, I occasionally get to hop into a terminal and tear out a few lines of code. While Ramona was a bit of a bust that needs to revisit the drawing board before its ready to leave the nest, Virgil has taken off. Virgil is something I’ve been doing in-between other tasks with the sole purpose of allowing users to edit and manage CodeLists managed in DDI. This is based on work I did mid-last year to turn DDI Code and Category Schemes into interactive webpages. To support this I’ve been working on a tool to allow users to properly edit Codelists in DDI.

A CodeList is a combination of two DDI objects, a CodeScheme and a CategoryScheme and enables users to manage complex hierarchies of coded information, as small as codifiying “Yes/No” responses to managing large industrial classifications.

To demonstrate how this may be done, I’ve uploaded a screencast of Virgil-UI in action opening a DDI version of the coded hierarchy from the Australian and New Zealand Standard Industrial Classification (ANZSIC) editing and saving the file.

The video demonstration is available on youtube – here.

The video got downscaled when it was uploaded (pressing the expand button helps) but for those having trouble understanding whats in the video, the features demo’d in the video are:

• Open the ANZLIC DDI File in the Vim text editor and searching for the term “LOOK HERE”. This search term isn’t in the file… yet
• Virgil-UI is run and the same file is loaded
• Data from the DDI File for a Category is loaded and is displayed in English and German
• The term “LOOK HERE” is added to the description of a category and the file is saved
• The file is then reloaded in the Vim text editor and the term “LOOK HERE” searched for
• The search term “LOOK HERE” is found

When ready (hopefully mid-August for open-beta) Virgil-UI will be released under an free open-source licence and will support the following features – ** Indicates a feature that is fully or partially implemented already
** Complete multilingual support, for both the UI and multilingual DDI files.
** DDI3.x file support
** Full rich-text editing for DDI Descriptions and Labels
** Support for Windows, Mac and Linux
* Export support for Virgil-Web an existing tool for generating Web-pages from DDI CodeLists
* Import from CSV
* Drag-and-drop re-ordering of CodeLists

Planned features after the initial release include:
* DDI2.x file support
* DDI3.x support from a custom-built repository
* DDI3.x support from a Colectica repository