Just a quick update that was supposed to have gone up last night. There is a video up on youtube now, showing of some of the more finalised features of Virgil-UI.

This shows three big features – CSV import, drag-and-drop reordering of classifications and multilingual support for editing. This means a classification with a multilingual component, for example a Canadian Industry Classification could have the English and French components edited simultaneously.

As stated in the last post, there should be a Windows binary release of a beta version of Virgil-UI and an updated version of the convertor tool should be released early September.

Prologue – A story of data discovery

Users routinely search for information on the web, and it is no different to any other for of discovery, be it digital or physical. To help cement this idea, consider this simple chain of events.

1. A website author adds tags and text to a page appropriate to the content
2. A user searches for terms, dates and locations in a search engine, such as Google or Bing
3. After getting the results, the user reads the descriptions of the appropriate sites to deciding which sites to read or ignore
4. The user clicks the link to a page they are interested in and reads more.

By going through these steps the user has been able to quickly narrow down a potentially millions of pages of information, and find the most appropriate one for their needs. Could you say that a user searching through your data could do the same?

The Data Discovery Cycle

Good metadata is about leading users through the Data Discovery Cycle – Discovery, Description and Identification, or DDI *. These are the three steps users go through to find the data they need. If you know where something is, you won’t search for it. If you don’t know what something is you won’t care where it is.

Discovery

Discovery metadata is the first piece of information that helps a user find what they want. When a user is at the Discovery stage of the Data Discovery Cycle they know what they want to find, but don’t know what or where it is yet. As a data provider, it is your role to help a user find what that is, and it starts by answering the five W’s that users may want to know about data. For example, a user may need to know some of the following to begin narrowing down the data:

• Who gave (or collected) the data?
• What data was gathered?
• When was the data collected?
• Where was the data gathered?
• Why was this data collected?

‘How’ has been left off this list because how something occurred can be very complicated or domain-specific, and Discovery metadata should be relatively standardised across domains. Also of note, is that what questions a user asks can be very specific, and although there are standard was to express this kind of information, building effective services over these standards comes down to understand who is trying to find your data.

An example of an excellent discovery metadata standard is something like Dublin Core. Using just 15 base fields, Dublin Core allows a provider to capture the essence of a piece of data, allowing widely differing pieces of information to be placed in one registry allowing users to find what they need across many different areas.

Description

Descriptive metadata is information that helps a user narrow down what they are trying to find. At the Description phase a user has begun narrowing the field of data, and is begins investigation specific data sources. Once a user has culled the whole field of data down to the a short list of related information they want, they can examine the descriptions of data that matches their criteria and can narrow the field of results even further.

Example of descriptive metadata that help at this stage includes:

• The name of the data (which is different from an indentifier)
• Brief and in-depth descriptions of the data
• Specific labels attached users may know the data as

This is the least machine actionable of all three steps, and the most likely to be in a written language. What is important, is that this information helps users get a better feel for what the data is about and allow them to cull or keep the data that is most relevant. Once a user has narrowed down what they need, they can move on to actually retrieving it.

Identification

At its simplest Identification metadata can be as simple as a single Uniform Resource Identifier or series of complex identifiers. No matter how identification metadata is managed, it is still able to pinpoint a single piece of information. Once a user has reached the Identification stage they have found what they want, and now need to be able to locate and retrieve the actual data. Previous pieces of metadata that assisted them search through data can take on new roles outside of the Data Discovery Cycle.

Probably the best identification standard was mentioned above, the humble URI or Uniform Resource Identifier. Standard, easily resolved, infinitely extendable and widely used.

What about everything else?

Don’t take this simple division of information to mean that everything else is unimportant. Metadata that doesn’t fall into the above roles shouldn’t be discounted. For domain specific reasons the lists of information above, can become what the user is after. However, for the process of helping a user go through the Data Discovery Cycle, everything else really does become less important, and the above distinction can help you narrow down what the is the most important information to help users search through a registry of information.

So what is the point?

The point is look at how specific metadata helps or hinders a users ability to find what they are after. Too much and they become overwhelmed with options, too little and it become too difficult to find what they need. Likewise, if users are restricted to specific values for their metadata they may misinterpret their meaning of the controlled vocabulary. But again, if they are given to much freedom, it may become impossible for anyone to find anything.

Epilogue – A story of discovery revisted

Lets revisit our earlier story and look at how this maps to the Data Discovery Cycle:

1. A website author adds tags and text to a page appropriate to the content
2. A user searches for terms, dates and locations in a search engine, such as Google or Bing (Discovery)
3. After getting the results, the user reads the descriptions of the appropriate sites to deciding which sites to read or ignore (Description)
4. The user clicks the link to a page they are interested in (Identification)

In summary, good discovery metadata is about finding the balance of information needed to help users find what they need with minimal effort and maximum results. However, ultimately this means understanding who your users, what they are trying to find, and how they want to search for data – but people can be a lot harder to understand that data I’m afraid.

* Although, not the DDI you might be thinking of which is a good metadata standard, but doesn’t explain how users search for their data. However, when it comes to standards that help statistical data providers describe their work, The Data Documentation Initiative is probably the best tool for helping providers make the necessary information to help users through the Data Discovery Cycle.

Apologies to Dijkstra for butchering that quote again , but the rage against <table>s is getting out of hand. Back in the dark ages, when it was impossible to get consistent HTML rendering across browsers and platforms someone decided to use the <table> element , originally designed to markup tabular data, to layout webpages.

Ever since that moment <table>s have been an unfairly shunned part of the HTML spec. When CSS became a stable and supported spec many people started screaming from the roof tops “Stop using <table> it ruins webpages”, and too many people have taken this to heart, so much that any time tabular data needs to be presented bizarre alternatives are used.

This was an example of a bad table of data that caught my eye a while ago:

A screenshot for posterity

Its supposed to give an idea of the cumulative revenue of iPhone Apps over time, but its hard to tell that as there is no title. Good data tells a story, good metadata explains that story. There is no metadata here to explain any of these numbers, here is what a search engine sees when they look at the HTML on that page:

Period ending.....Period downloads.....Cumulative downloads....Period revenues
Jun 2008............no apps...................no apps........................no revenues
Dec 2008.............600 M......................600 M..........................$ 172 M
Jun 2009..............800 M....................1.4 B.............................$  228 M
Dec 2009..........1.6 B.........................3.0 B............................$  458 M
Jun 2010...........2.0 B.........................5.0 B............................$  542 M
Total.................5.0 B.........................5.0 B............................$1.4 B

Accessibility and search robots aside, here is no context to any of this, the units change between rows, the number of dots changes each time. Its hard to say what this data is without purposefully reading the text around it. I could drone on about the lack of metadata describing the data in this table, but instead I’ll counter this with a better example.

Here is the same table, rewritten as an actual <table>, using only what’s defined in the HTML4 and above W3C specifications:

A few things instantly stand out: the columns all match up nicely, both in display and in the table-model; each of the years is clearly marked; the table is relatively self-explanatory; and lastly it appears as a table to anyone who reads it. Furthermore, all of the formatting is done using CSS; thats right the presentation is left to the CSS and the descriptive markup is done using a <table>, just how it is intended to be done. But even this is only a fraction of the possible metadata to place into a HTML table.

However, this is not the worse example of a ‘table’ I have seen, the one that sent me over the edge was this large image that highlights the relative pros and cons of different smart phones:

This 'table' is too big to show inline, it needed to be reduced to a third of its size to fit

The main issue with this, is this image of a table is mostly text, is quite large and static. You cannot easily copy text out of this, you can’t easily reorder the table and unless you have a large screen it would be quite difficult to compare two products that weren’t both very close to each other. Unlike the table of iPhone revenues, I am not going to go to the effort of transcribing this into a proper HTML table.

Compare this with a similar table from wikipedia also providing a comparison of smartphones and the differences are obvious. First of all, the table is now web-crawlable – whatever data is in this table is now indexed in Google, instant bonus, and the user can easily search through the page to find what they needed to know. There is also a whole lot of Javascript on the page. For example, clicking the boxes next the each of the column titles reorders the whole table around that column. In the first table this isn’t too noticeable with most entries having either ‘Yes’ or ‘No’, but the Hardware and OS table from further down the same page is full of figures, and now I can easily find the lightest phone (it’s the HP iPAQ Voice Messenger, something that would be much harder to find out on the image ‘table’).

So in the midst of preparing for Cornell, Uni work and wedding planning I haven’t had time to update, but taking a breather in a beautiful hotel room is a perfect time to get back to blogging.

I arrived in Ithaca yesterday, the trip up way gorgeous. I would say pun intended, but I didn’t know Ithaca was famous for it gorges until I got here. Spend the afternoon with folks from the DDI Alliance Expert Committee. That was great, didn’t say much, but listening to some extraordinarily smart people and having them do the same to me was honouring. Sadly after dinner I had to retire with a killer fog of jetlag and nearly immediately fell asleep.

This morning was much more eventful, the lovely staff at the Statler Hotel were able to get me a cord to charge my camera with, and with that I set off to explore the campus and surrounds. I walked round the campus before setting off for Collegetown. As my camera only had a short charge all my shorts of Collegetown are on my phone, whose cord is absent, so until I find it or figure out how to get the shots off, no photos from there yet.

But thats ok, as the best shots were down by Casadilla Gorge. The shots were great, and by the bend down the end there is (according to locals) a little wading area. The track along the creak from there was spectacular and green after last nights rain.

The only downside is a twisted my ankle some time during the walk, but I’m all strapped up, about to head of for lunch and see Jeremy Iverson from Algenta talk about Colectica.