Drivel about data and musings about metadata.

The latest Australian CPI figures were released last week by the Australian Bureau of Statistics.

Fortunately, each release includes re-weighted indices for each indicator. These cover 11 major topics, including food, clothing, housing and education. Unfortunately, the Excel spreadsheets these are distributed aren’t the easiest formats to process data from. This is because exporting data from Excel can be time consuming, and the data as it is stored in Excel neglects the hierarchical structure of the CPI indicators.

To help change this, I wrote a few scripts (and lovingly hand-crafted some XML) to help transform the CPI from Excel into a DSPL dataset, and have uploaded this into into the Google Public Data Explorer.

The dataset that has been uploaded is based of Table 12 from the Downloads section of the latest CPI page. This dataset includes the indices for each capital city, and Australia, for each level of indicator – from the broad total CPI to, for example, the more specific “Food”, “Bread and Cereals” and finally “Breads”. There are 12 total broad topic covered, including a miscellaneous group of indices that exclude some of the 11 topics, and there are 144 topics at the finest detail.

The end result is something like this:

If you want to play with the whole dataset, it is available on the Google Public Data Explorer, or if you would like to download the full DSPL dataset, that is available on the DSPL-R downloads page.

Probably one of the more interesting parts was how to create the hierarchical CPI indicators category in DSPL, but I’ll be following this post up later in the week with a tutorial on how to work with complex datasets.

Update: With the help of a kind statistician from Google the datset is now much better structured. The updated dataset is available here: http://code.google.com/p/dspl-r/downloads/list

It was recently announced that Google has made their Public Data Explorer open to the public, so now anyone can upload data. While the data that they have made available is interesting, what is more interesting is the much subtler announcement of their DataSet Publication Language (DSPL).

DSPL is a data/metadata language specification that basically allows people to describe multi-dimensional, aggregate datasets, along with their appropriate metadata in a structured way. For those of you playing along at home, this is almost identical to the ideas behind SDMX. Both languages support datasets by essentially defining compound keys of dimensions and their associated measures – dimensions and metrics in DSPL and key families and measures in SDMX. However, there are three significant differences between the two standards which will impact which one will see the wider adoption.

1: SDMX was a collaborative effort to meet the needs of a wide banking and statistical community, DSPL was made solely by Google to accommodate the Public Data Explorer.
For good or bad, when making DSPL, Google only had to meet their own needs: design a standard that will help people write data for the Public Data Explorer, it would be good if it was easy. SDMX on the other hand, has had a lot of hands pulling it, which means the standard, as well written as it is, is overly complex and meets more needs than any one agency could every likely encounter. The important thing about the size of the spec, is that it has a near linear relationship to the size of the documentation, which means…

2: DSPLs documentation is vastly shorter than SDMX.
When I first began looking at SDMX, the documentation was enormous. To be able to go through the simple task of making a dataset, and adding its metadata required reading through reams of documentation. In the end it was possible, but not without a significant investment of time, and for many people if the time to produce something productive is too long they will start to look elsewhere. However, the documentation for DSPL is light, and easy to understand, such that it took me much less time to start working with the standard productively *. However, one of the main reasons the documentation is so small is because…

3: SDMX manages all data and metadata in XML, DSPL uses XML and CSV
This leads to a huge difference in understanding how to start working with data. DSPL works for the same idea as SDMX that data should be defined according to its dimensions, and that these dimensions should be described as XML to allow people and machines to understand them. However, DSPL manages all its data, ie the actual numbers, as tabular in Comma Separated Value (CSV) files, while SDMX serialises the data in XML constructs – this is a massive difference.

Any programmer worth their salt can process CSVs: readline, split by commas, do things with values, repeat. There are standard libraries for managing CSVs, it can be imported into most RDBMS and is almost a defacto standard for data – even Excel supports it. This means that to write a tool to work with DSPL, one only needs to learn one new thing, rather than two. It also acts as an improvement to an existing technology, rather than a completely new idea.

4: DSPL is backed by Google, SDMX isn’t
This is probably the biggest difference of all. Despite the fact that SDMX may be managed by some pretty heavy weight players, weighing in against Google they look like small fry. Google also has the advantage that it has visibility with developers and that will make the big difference between the two. A standard could be the best designed, easiest to learn, and most useful, but if no-one is writing tools for it, if no-one is promoting it, if there aren’t the resources to work with it, then it is doomed to fail. Google has the ability, experience and resources to drive this ground swell.

So where does this leave our two standards?

Well, hopefully, it leaves them in a position to work with each other. DSPL acts as a great entry point into the more complex SDMX metadata requirements – an SDMX-Lite almost. This is especially of note, considering that with such a large overlap, it would take little effort to write a tool to convert data between the two standards – albeit with a lossy conversion between SDMX and DSPL.

However, for either to ignore that the other exists is a dangerous proposition. With each format having the potential to tap into slightly different markets, a division between them would present a split between people who ultimately have similar goals,.

* One could argue that I was able to understand DSPL so quickly because I had already learned how SDMX worked. One wouldn’t be wrong, but the point about the difference in documentation holds.

NB: This post uses examples of Unicode that may not show up in some browsers.

One of my main gripes with twitter is the ability to add only text. People often have the desire to share small snippets of data, but to no avail. The ideal idea to share data in such tiny chunks of data Edward Tufte idea of a Sparklines.

For those of you disinclined to read the wikipedia page, sparklines are “data-intense, design-simple, word-sized graphics”, designed to be entered inline with text, at similar height to help illustrate an idea.

Now I am not the first person to suggest entering sparklines in to twitter, in fact the second entry for a google search for sparkline turns up Alex Kerin’s article. However, there are two slight problems with Kerin’s implementation. Firstly, the unicode block characters he is using are not designed to be lined up, and examples that are shown on his page demonstrate this. To be fair, this isn’t his fault at all as unicode compliance isn’t 100%. The second is that a bar and a line can provide two very different perceptions: bar charts generally being used to display discrete data (or continuous data being shown as discrete) and line charts being used to show continuous data – for the record there is no good time to use a pie chart.

To this end I have created a tool for producing two different types of sparkline from an input data source – A crude line graph and a 5-figure box-plot.

Here is an example showing this are using the June 30th 2010 Perth weather data from the Bureau of Meterology, with bars delimiting 3 hour blocks:

The weather yesterday in Perth was quite cool (4.1┣▇▇|▇━━┫17.7) with a maximum of 17.7 degrees occuring around 2pm, before quickly cooling down until 3pm. (⣤⣤⣀⎸⣀⣀⣀⎸⣀⣀⡤⎸⠴⠚⠛⎸⠛⠛⠙⎸⠒⠒⠒⎸⠒⠲⠶⎸⠶⠶⠶).

Limiting this example further, restricting ourselves to the 140 characters of twitter:

Perth 30/06/10: Cool (4.1┣▇▇|▇━━┫17.7), max at 2pm, cooling to around 13°C after 3pm, steady afterwards. (⣤⣤⣀⎸⣀⣀⣀⎸⣀⣀⡤⎸⠴⠚⠛⎸⠛⠛⠙⎸⠒⠒⠒⎸⠒⠲⠶⎸⠶⠶⠶)

This is a 115 character weather report leaving 25 characters for a url to the full data. This may be for temperature only, but it shows the potential and can place 2 dataset in a twitter post with commentary.

I think the boxplots look quite good, however the tool does take a few liberties with the braille layout, relying on people to see a pair of vertical dots as a value in between the two, but it helps convey the message quite well in a limited, text-based format.

So, I’m sitting in the Sydney Qantas Business Lounge, with plenty of time until my connecting flight home, so I’ll recap the last few days.

From what people had to say my presentation went quite well. The script I worked from is now available online, and the slides will soon follow. As soon as I have updated the slides with the transcript I will upload them and update this post.

There were however plenty of excellent talks at the IASSIST Conference this year. Due to the theme there was an abundance of social networking talks, including the keynote speaker on Friday, and they all presented excellent angles on the same question: “How can we get more people interacting with our data better?”

I think this was a great take on where data and metadata agencies are heading, as as I have said before, due to the fact that the more we have people interacting with data agencies, and using the data and metadata they provide, the more relevant those agencies become. The fact there are so many agencies asking this question and working on their own solutions and are willing to share those ideas is a great step forward for open data.

Over the past year there has been a global shift in data policies within governments world-wide – opening up data, that once cost or was hidden from public view. Some government organisations have gone so far as to put together incentives to encourage people to make use of this newly freed data. Australian examples include state initiatives such as Victoria’s AppMyState and New South Wales’ apps4NSW and the Federal Governments Government 2.0 taskforce’s Mashup Australia. What’s important to realise though is that these promotions should be seen more as a means to an end, rather than an end in and of themselves.

Data collectors and maintainers need to exhibit a relatively low level of bias and high levels of independence. If any data collected is called into question, the validity of the entire archive, and future collections, can also be questioned. The reason for this stems for the logical (albeit poorly grounded) assumption that if one collection had been tampered with then there are grounds to suggest that all other collections may suffer the same bias.

It is for this reason that the data exposed from these sources may be little more than the aggregated and weighted data, with explanatory metadata and appropriate notes on the methodology of the study. No exposition into the links between data, or overtly controversial or political hypotheses attempting to explain the data. Anything more may suggest bias or influence: for example, a statistical agency may present statistics regarding life expectancy in collection regions, and in another study my present statistics around polluting industry in the same regions, but would be amiss to begin to draw correlations between them. The same agency may hold for data around economic indicators, but would never try to correlate them to political policy.

So while the agency may hold this data and the expertise in explaining the data, they would be reluctant to go further than collection and dissemination. However, the public is unencumbered by such ideals.

Contests such as those listed above give public agency the opportunity to drive public use of public data. This again drive straight to the goals of data agencies: public use of data reenforces their relevance. This gives the public the opportunity to ask potentially controversial questions, backed by official data, while giving the data agencies recognition for their continued importance in society.