Drivel about data and musings about metadata.

1. Do you live with mommy and daddy? 
2. Does mommy or daddy smoke?
3. Do you eat a lot of fruit or nuts at home?

The Australian Bureau of Statistics recently released data cubes on Household Expendituregiving an incite into how Australians use their money. What is of interest is that they provide a breakdown of average weekly expenditure for a variety of products, tabulated against the number of financial stressors. The ABS has defined financial stressors are events where a household is unable to pay bills, goes without meals,

The reason this is interesting is that there are very few factors with a striking correlation to the number of financial stresses within a household as those below:

Note, that for newspaper and fruit and nut weekly expenditure, there is a negative correlation.

The problem with some of these metrics though is while they are all strongly correlated with financial stress, they may not all be obvious to children. For example, a child may not know about their parents income or if they live in a rental property, or certain actions may not be hidden from the child, such as a parent buying a newspaper on the way to work.

Other metrics however, are more obvious for children to notice and report. Such who they live with, obvious activities of their parents,  (like smoking) and their own diets. This leads us to the three questions listed above:

The following is a letter I wrote, with the intention to forward it to the heads of Statistics New Zealand and those behind the decision to cancel the 2011 New Zealand Census, however I think it rings true for all concerned about the future of the Census in all nations.

————————–

The earthquakes that recently devastated Christchurch were grave, terrible events and while New Zealand and its close neighbour Australia mourn for the loss of life, I implore you: halting the census will not ease this grief. In fact, it will have a much graver impact on the statistical world, outside our small Pacific community.

I do not need to educate any of the intended recipients of this letter about the role of the census, but for completion I feel it is necessary to remind all how vital the census is to the community.

The data from a census provide nations with the ability to better understand their past and present and dictate their future. Each census is a fully comprehensive snapshot of the entire population; by comparing them, we reflect on how nations change over time. Studying how demographics evolve, we as a society are able to create public policy that ensures all citizens are represented and provided the resources they need to thrive. Census information helps to create fair electoral boundaries, encourages the creation of schools, hospitals and roads where they are needed by the community, and even assists medical research. There is a reason censuses have been going on for thousands of years, and why the establishment of a census bureau is one of the first actions of a just and educated nation.

While the role of these agencies may have been clear at first, over time citizens are forgetting how the census benefits them more than it benefits the state. Both national censuses and the agencies that run them are already under threat from many parties. There are those who seek to cripple the census, either to superficially save tax dollars or alter the demographic information to better skew their positions. The government of the Right Honourable Stephen Harper, Prime Minister of Canada, recently made changes to the 2011 Canadian Census to make portions of the census voluntary. This move was surrounded by controversy and outcry from the statistics community. Much of this controversy focused on the methodological issues a voluntary ‘census’ would have on the data and how this would impact the community, as well as acting as a direct challenge by the conservative government to the agency’s independence.  Ultimately, this led to the resignation of Mr. Munir Sheikh, then Chief Statistician of Canada, as an act of protest regarding the autonomy of Statistics Canada.

The actions of those who wish to harm our profession are not confined to Canada. In the United States of America, the 2010 Census came under fire from political conservatives over its relevance, including calls from influential political icons to call for boycotts of the census. These critiques ultimately ended in the removal of the “long form”, which may lead to immeasurable problems around the reduced data quality of the census. Likewise, in the past decade, the Office of National Statistics in the United Kingdom has faced similar issues over its autonomy and independence from government, leading community leaders to question its impartiality as a statistics provider.

The cancelling of the 2011 New Zealand Census will be the first cancelled census by New Zealand since World War Two, and joins a very short list of cancelled censuses by a Commonwealth nation in the same time. While the act of cancelling the 2011 census is an act of compassion, we run the risk of those who wish to call for an end to the census to use this tragedy as an example of why census is no longer a required instrument of a just and democratic society. I assure you that the census is no less vital today than it was in 1911 when the Australian public was first counted, and it will still be relevant in 2051 when New Zealand marks its 200 years of census.

The statistics bureaus of Australia and New Zealand hold a special position amongst Commonwealth and anglophone countries, as two of the most autonomous official statistical agencies, but this comes at a price; we should look upon ourselves as role models in the statistical community and go as far as it takes to ensure our independence and relevance, but above all the unquestionable quality of our data. While their are deep emotional reasons for cancelling of the census, based purely for the respect of those most hurt by the tragic events of the Christchurch earthquakes, sometimes we must look at the rational reasons to push ahead with painful policies.

Right now Christchurch is in mourning and recovery, but soon it will be time to rebuild your great city. The reconstruction works will employ many of those put out of work by the disasters due to loss of lives, homes or business, but will not employ them all. By running the census you are able to employ additional thousands to prepare for its great counting, thousands who may not have otherwise been employed. These will be temporary positions that will allow people to immediately begin rebuilding their lives and city. Rather than just receiving government benefits to help the rebuilding, you will be employing people in stable short-term work to allow them to begin the road to normal lifes again, work that will benefit them and their country.

News reports are already indicating that Statistics New Zealand will honour its contacts and agreements with those employed throughout the census. If the decision has been made to spend this money, then spend it helping your fellow countrymen. Not only will you be employing those who may have been jobless, but you are also effectively helping to push money into a damaged economy. Each census worker will need shelter, food, clothes—all things that can be bought within their town, pushing more funds into the area, and speeding the recovery.

With the Australian people also due to run a census in 2011, the Australian Bureau of Statistics is already in preparations for what is said to be “Australia’s largest peacetime operation”. In peace, as in war, ANZACs should stand shoulder to shoulder, ready to help each other regardless of our mission or the circumstances. Australia as a nation have already given much to help our Tasman neighbours, and we should stand ready to give more. With infrastructure in place for several weeks prior to the additional census, and with so many standards designed in collaboration between our nations, Australia is more than ready to help you perform your civil duties.

Lastly, as with the census in Australia being so close to the flooding in Brisbane, running a census in the shadow of a grave disaster such as the one that befell you, will help us measure the damage that has occurred and will improve future estimates of damage. Performing this census will allow communities to understand better the damage that can befall them, and to better prepare them for the worse.

Honourable readers, Prime Ministers, Commonwealth Statisticians and fellow citizens, the census is not just an exercise in counting, it is the snapshot of a nation and the prime way that we as a nation can better understand ourselves and our place in the world and can’t be replaced by a mere sample. It is one of the most noble acts of a society as we make every citizen count. In a census, all are equal, the rich or the poor, the young or the old, every minority enumerated for all to see. It is the candle in the darkness that shines a light on society and exposes our weaknesses and triumphs. But there are those who wish to extinguish this flame of impartiality for their own ends, they are those who wish to hide the inequalities of life, shun the impoverished and disenfranchised and maintain the bigotry that hurts us all.

When we are besieged by such evil forces, we cannot falter, and by cancelling the New Zealand census, you give these malcontents the fuel they need to further jeopardise the ability for all statistical agencies to fulfill their vital role. If you continue with this action you harm us all, and I plead of you, continue the census. Give the census a firm date and give the people of your nation a glimmer of light and show them that not even shattered earth will stop a statistician’s resolve. Those who survived may count themselves lucky, without your enumeration their efforts will be for nought.

NB: This post uses examples of Unicode that may not show up in some browsers.

One of my main gripes with twitter is the ability to add only text. People often have the desire to share small snippets of data, but to no avail. The ideal idea to share data in such tiny chunks of data Edward Tufte idea of a Sparklines.

For those of you disinclined to read the wikipedia page, sparklines are “data-intense, design-simple, word-sized graphics”, designed to be entered inline with text, at similar height to help illustrate an idea.

Now I am not the first person to suggest entering sparklines in to twitter, in fact the second entry for a google search for sparkline turns up Alex Kerin’s article. However, there are two slight problems with Kerin’s implementation. Firstly, the unicode block characters he is using are not designed to be lined up, and examples that are shown on his page demonstrate this. To be fair, this isn’t his fault at all as unicode compliance isn’t 100%. The second is that a bar and a line can provide two very different perceptions: bar charts generally being used to display discrete data (or continuous data being shown as discrete) and line charts being used to show continuous data – for the record there is no good time to use a pie chart.

To this end I have created a tool for producing two different types of sparkline from an input data source – A crude line graph and a 5-figure box-plot.

Here is an example showing this are using the June 30th 2010 Perth weather data from the Bureau of Meterology, with bars delimiting 3 hour blocks:

The weather yesterday in Perth was quite cool (4.1┣▇▇|▇━━┫17.7) with a maximum of 17.7 degrees occuring around 2pm, before quickly cooling down until 3pm. (⣤⣤⣀⎸⣀⣀⣀⎸⣀⣀⡤⎸⠴⠚⠛⎸⠛⠛⠙⎸⠒⠒⠒⎸⠒⠲⠶⎸⠶⠶⠶).

Limiting this example further, restricting ourselves to the 140 characters of twitter:

Perth 30/06/10: Cool (4.1┣▇▇|▇━━┫17.7), max at 2pm, cooling to around 13°C after 3pm, steady afterwards. (⣤⣤⣀⎸⣀⣀⣀⎸⣀⣀⡤⎸⠴⠚⠛⎸⠛⠛⠙⎸⠒⠒⠒⎸⠒⠲⠶⎸⠶⠶⠶)

This is a 115 character weather report leaving 25 characters for a url to the full data. This may be for temperature only, but it shows the potential and can place 2 dataset in a twitter post with commentary.

I think the boxplots look quite good, however the tool does take a few liberties with the braille layout, relying on people to see a pair of vertical dots as a value in between the two, but it helps convey the message quite well in a limited, text-based format.