1. Do you live with mommy and daddy? 
2. Does mommy or daddy smoke?
3. Do you eat a lot of fruit or nuts at home?

The Australian Bureau of Statistics recently released data cubes on Household Expendituregiving an incite into how Australians use their money. What is of interest is that they provide a breakdown of average weekly expenditure for a variety of products, tabulated against the number of financial stressors. The ABS has defined financial stressors are events where a household is unable to pay bills, goes without meals,

The reason this is interesting is that there are very few factors with a striking correlation to the number of financial stresses within a household as those below:

Note, that for newspaper and fruit and nut weekly expenditure, there is a negative correlation.

The problem with some of these metrics though is while they are all strongly correlated with financial stress, they may not all be obvious to children. For example, a child may not know about their parents income or if they live in a rental property, or certain actions may not be hidden from the child, such as a parent buying a newspaper on the way to work.

Other metrics however, are more obvious for children to notice and report. Such who they live with, obvious activities of their parents,  (like smoking) and their own diets. This leads us to the three questions listed above:

With the recent release of the new Australian Standard Classification of Drugs of Concern from the ABS, there was the opportunity to field test the Virgil CSV to DDI converter with real data to see how it held up. Fortunately, the classification was released as an Excel data cube that conformed almost entirely with the structures that Virgil supports. After a little cleaning of the CSV, it was able to run through the converter without few issues at all. Incidentally the most major error highlighted the massive oversight that the converter fails to add values for the codes! However this has been corrected and changes have been pushed in the svn, and a new version of the Windows tool will be pushed out this weekend.

A screen shot of Virgil with the converted classification

Opening the newly created DDI file in the Virgil DDI CodeList Editor was another story and pointed out a few flaws with how it handles empty data. With the structure from the Excel file not containing descriptions for any category or any labels for the CodeScheme, there were a few small corrections made to accommodate freshly created DDI, but many of these problems will be ironed out by the time the CodeList editor is available for download.

While the converter hasn’t been fully integrated into the CodeList Editor, it will shortly be possible to create a single DDI file and import numerous CSV files to create a series of classificatory codelists in a single package. A practical and soon to be realised example would be the Australian Standard Classification of Drugs of Concern with the lists of drugs of concern, forms of drug and methods of consumption codelists all contained in a single machine processable DDI package.

For those who haven’t been able to download or run the converter, the output from this example is available for testing.

This is the second in a 5 part series of working with questionnaires and surveys managed using the Data Documentation Initiative XML standard. With DDI being an emerging technology, it is important that users are provided with best practises to ensure that they use the standard in a way that is logical, coherent and most importantly usable and reusable. This series of tutorials and discussions is aimed toward users who have some knowledge of DDI and would like to know how to effectively design markup existing and future questionnaires in DDI.

Part 2 : Where am I and where do I go next?

In the previous post on questionnaires using DDI we looked at how it is possible in DDI to repeat sequences of questions, potentially leading to endless loops of questions within a survey. In a paper survey this would be quickly picked up, but in an electronic survey this might not always be the case. To help identify these issues  a tool called Sheri was introduced that consumes a DDI instrument and check for potential issues of repeated questions and endless loops. Sheri also is able to generate a non-standard XML format representing an entire survey marked up in DDI. But this leads to the question “if one were to design a survey in DDI, why introduce another XML format?”

During research into DDI as a data capture standard it became quickly apparent, that for all of its benefits in designing surveys, DDI was very difficult for programs to consume to create instances of these instruments. This is due to how DDI describes an Instrument. In DDI a whole survey instrument is represented by a single tag – Instrument -, which in turn references a single Sequence within a ControlStructureScheme. This is simple enough, to understand, but once the first Sequence is found the structure becomes quite interesting. Every Sequence, or other ControlConstruct, maintains references to its child sequences. For example, a Sequence can reference multiple other Sequences, which reference more Sequences and so on. Likewise, a Loop maintains a list of linked ControlStructres to loop over, and conditional IfThenElse tags contains references in both Then and Else clause that refer to other ControlStructures. What this means is that although there is an implied hierarchy, it is stored as a flat list of data structures with references between them.

What this leads to is a difficulty in determining where exactly in the hierarchy one is if they are given only the id of the current structure, especially when dealing with the stateless nature of the web. For example, the test software I was writing, Ramona, would take a sequence id as an argument and render the corresponding DDI sequence as a form on a webpage with web controls for each question. What quickly became an issue was determining the what the next page to display was when a user was done with a sequence.

In a DDI ControlStructure, the ID of a child element is stored as the text of an ID element under a ControlConstructReference. What this means is that to determine the next possible sequence given a sequence id, you need to look for references to the object, not the object itself. Then from the reference, search back for an appropriate ancestor element and then return the next sibling to show the correct form. This quickly becomes complicated and when dealing with large DDI test  files doing plain text searches throughout an unmanaged * hierarchy it became apparent that this method was too slow and complex for real time returns.

A further complication arises if a survey reuses an object using multiple references (as opposed to using Loops). In cases such as this, using the above method it becomes impossible to easily determine which is the correct parent. The reason being that a reverse lookup for referring objects for an object referenced multiple times will only return a list of referring objects with no context about which one we need to follow.

For example in the following sequence:

Sequence id="Seq 1"
    sequence reference: Seq 3
Sequence id="Seq 2"
    sequence reference: Seq 3
Sequence id="Seq 3"
    question reference: Q1: Where did I come from?

Resolves to have a structure like:

Sequence id="Seq 1"
    Sequence id="Seq 3"
        question reference: Q1: Where did I come from?
Sequence id="Seq 2"
    Sequence id="Seq 3"
        question reference: Q1: Where did I come from?

But, given just the id of Sequence 3 we can’t determine if after answer the question the survey is over (if we arrived at Sequence 3 from Sequence 2), or if we still have to go to Sequence 2.

To solve these issues of both speed, development and determining location, it is therefore necessary for applications using DDI to “pre-compile” Instruments into a traditional hierarchy form. In both Ramona and Sheri, the solution is to resolve the references and copy the referenced elements into the parent structure. This allow much of the DDI metadata to be retained and used.

This is not valid DDI, and it is unlikely that it ever will be.

This is also not a problem. DDI is useful as an archival and transportation language for statistical metadata, and the flexibility that the current structure provides is quite useful. However, when looking at data collection, it can be safely assumed that the DDI Instrument would be relatively stable. If best practices are followed, once a DDI Instance is published it will never change. Thus it is a perfectly valid action to transform an Instrument in this way, as long as two conditions are met: this is seen as a one-way destructive transformation, and that the resulting pseudo-DDI instrument is never changed. To provide another example of why this is a normal situation, there are tools that are being developed to transform DDI into PDF questionnaires: this is very much the same process, the PDF is seen as a projection of the original DDI to make it easier for people to use, but not the actual ‘source-of-truth’. Transforming DDI Instruments into a dereferenced pseudo-DDI Instrument is exactly the same, a transform of complex metadata into a form that machines can easily work with.

There is one issue that these pseudo-DDI Instruments will have, and that is when a single data structure is referenced multiple times, it will occur multiple times in the resultant tree. In cases like this, it is still difficult to determine which element is the correct one when just given an ID. There are two possible solutions to this issue, the first being that instead of managing state based on a single ID, it is managed as the full XPath of the element, possibly speeding up traversal, but also presenting the possibility the the structure of the form could be shared with users – which may or may not be a security issue depending on the form. Alternatively, as discussed in the part one of these tutorials, restructuring Instruments so no structure is referenced more than once, making it easier to traverse through the form, as well as limiting user frustration.

It should be noted that restructuring is not a necessity in making DDI Instruments processable by software, just something that makes them easier to use using traditional methods and existing XML libraries, and can provide benefits to execution times if that is an issue. However, when dealing with desktop software, many of the issues of web development with regards to stateless vs. stateful systems or the scalability of systems with concurrent users cease to be an issue. In such situations, it is quite possible to work using the DDI Instrument directly, with manipulating the data strucutre, and in some cases may be preferable.

In conclusion, how strictly we manage the DDI metadata structure depends very strongly on the role the metadata plays in a system. In some cases, such as computer-aided interviewing where the instrument should be extremely stable, the transformation of DDI to a format that is more easily processed by systems, or even users, can be preferable to using plain DDI. what is important is to focus on DDI as a tool for increasing transparency and reusability in statistical processing, but as long as the methods used to transform DDI into intermediate forms are well documented there is no reason why this cannot be done as such the use of well-documented transformations would not violate existing best practice.

* Unmanaged in the sense that the hierarchy is stored as references between XML elements, and not as a traditional XML hierarchy, and as such traditional tree traversal methods for XML cannot be used.

The latest Australian CPI figures were released last week by the Australian Bureau of Statistics.

Fortunately, each release includes re-weighted indices for each indicator. These cover 11 major topics, including food, clothing, housing and education. Unfortunately, the Excel spreadsheets these are distributed aren’t the easiest formats to process data from. This is because exporting data from Excel can be time consuming, and the data as it is stored in Excel neglects the hierarchical structure of the CPI indicators.

To help change this, I wrote a few scripts (and lovingly hand-crafted some XML) to help transform the CPI from Excel into a DSPL dataset, and have uploaded this into into the Google Public Data Explorer.

The dataset that has been uploaded is based of Table 12 from the Downloads section of the latest CPI page. This dataset includes the indices for each capital city, and Australia, for each level of indicator – from the broad total CPI to, for example, the more specific “Food”, “Bread and Cereals” and finally “Breads”. There are 12 total broad topic covered, including a miscellaneous group of indices that exclude some of the 11 topics, and there are 144 topics at the finest detail.

The end result is something like this:

If you want to play with the whole dataset, it is available on the Google Public Data Explorer, or if you would like to download the full DSPL dataset, that is available on the DSPL-R downloads page.

Probably one of the more interesting parts was how to create the hierarchical CPI indicators category in DSPL, but I’ll be following this post up later in the week with a tutorial on how to work with complex datasets.

Update: With the help of a kind statistician from Google the datset is now much better structured. The updated dataset is available here: http://code.google.com/p/dspl-r/downloads/list

The following is a letter I wrote, with the intention to forward it to the heads of Statistics New Zealand and those behind the decision to cancel the 2011 New Zealand Census, however I think it rings true for all concerned about the future of the Census in all nations.

————————–

The earthquakes that recently devastated Christchurch were grave, terrible events and while New Zealand and its close neighbour Australia mourn for the loss of life, I implore you: halting the census will not ease this grief. In fact, it will have a much graver impact on the statistical world, outside our small Pacific community.

I do not need to educate any of the intended recipients of this letter about the role of the census, but for completion I feel it is necessary to remind all how vital the census is to the community.

The data from a census provide nations with the ability to better understand their past and present and dictate their future. Each census is a fully comprehensive snapshot of the entire population; by comparing them, we reflect on how nations change over time. Studying how demographics evolve, we as a society are able to create public policy that ensures all citizens are represented and provided the resources they need to thrive. Census information helps to create fair electoral boundaries, encourages the creation of schools, hospitals and roads where they are needed by the community, and even assists medical research. There is a reason censuses have been going on for thousands of years, and why the establishment of a census bureau is one of the first actions of a just and educated nation.

While the role of these agencies may have been clear at first, over time citizens are forgetting how the census benefits them more than it benefits the state. Both national censuses and the agencies that run them are already under threat from many parties. There are those who seek to cripple the census, either to superficially save tax dollars or alter the demographic information to better skew their positions. The government of the Right Honourable Stephen Harper, Prime Minister of Canada, recently made changes to the 2011 Canadian Census to make portions of the census voluntary. This move was surrounded by controversy and outcry from the statistics community. Much of this controversy focused on the methodological issues a voluntary ‘census’ would have on the data and how this would impact the community, as well as acting as a direct challenge by the conservative government to the agency’s independence.  Ultimately, this led to the resignation of Mr. Munir Sheikh, then Chief Statistician of Canada, as an act of protest regarding the autonomy of Statistics Canada.

The actions of those who wish to harm our profession are not confined to Canada. In the United States of America, the 2010 Census came under fire from political conservatives over its relevance, including calls from influential political icons to call for boycotts of the census. These critiques ultimately ended in the removal of the “long form”, which may lead to immeasurable problems around the reduced data quality of the census. Likewise, in the past decade, the Office of National Statistics in the United Kingdom has faced similar issues over its autonomy and independence from government, leading community leaders to question its impartiality as a statistics provider.

The cancelling of the 2011 New Zealand Census will be the first cancelled census by New Zealand since World War Two, and joins a very short list of cancelled censuses by a Commonwealth nation in the same time. While the act of cancelling the 2011 census is an act of compassion, we run the risk of those who wish to call for an end to the census to use this tragedy as an example of why census is no longer a required instrument of a just and democratic society. I assure you that the census is no less vital today than it was in 1911 when the Australian public was first counted, and it will still be relevant in 2051 when New Zealand marks its 200 years of census.

The statistics bureaus of Australia and New Zealand hold a special position amongst Commonwealth and anglophone countries, as two of the most autonomous official statistical agencies, but this comes at a price; we should look upon ourselves as role models in the statistical community and go as far as it takes to ensure our independence and relevance, but above all the unquestionable quality of our data. While their are deep emotional reasons for cancelling of the census, based purely for the respect of those most hurt by the tragic events of the Christchurch earthquakes, sometimes we must look at the rational reasons to push ahead with painful policies.

Right now Christchurch is in mourning and recovery, but soon it will be time to rebuild your great city. The reconstruction works will employ many of those put out of work by the disasters due to loss of lives, homes or business, but will not employ them all. By running the census you are able to employ additional thousands to prepare for its great counting, thousands who may not have otherwise been employed. These will be temporary positions that will allow people to immediately begin rebuilding their lives and city. Rather than just receiving government benefits to help the rebuilding, you will be employing people in stable short-term work to allow them to begin the road to normal lifes again, work that will benefit them and their country.

News reports are already indicating that Statistics New Zealand will honour its contacts and agreements with those employed throughout the census. If the decision has been made to spend this money, then spend it helping your fellow countrymen. Not only will you be employing those who may have been jobless, but you are also effectively helping to push money into a damaged economy. Each census worker will need shelter, food, clothes—all things that can be bought within their town, pushing more funds into the area, and speeding the recovery.

With the Australian people also due to run a census in 2011, the Australian Bureau of Statistics is already in preparations for what is said to be “Australia’s largest peacetime operation”. In peace, as in war, ANZACs should stand shoulder to shoulder, ready to help each other regardless of our mission or the circumstances. Australia as a nation have already given much to help our Tasman neighbours, and we should stand ready to give more. With infrastructure in place for several weeks prior to the additional census, and with so many standards designed in collaboration between our nations, Australia is more than ready to help you perform your civil duties.

Lastly, as with the census in Australia being so close to the flooding in Brisbane, running a census in the shadow of a grave disaster such as the one that befell you, will help us measure the damage that has occurred and will improve future estimates of damage. Performing this census will allow communities to understand better the damage that can befall them, and to better prepare them for the worse.

Honourable readers, Prime Ministers, Commonwealth Statisticians and fellow citizens, the census is not just an exercise in counting, it is the snapshot of a nation and the prime way that we as a nation can better understand ourselves and our place in the world and can’t be replaced by a mere sample. It is one of the most noble acts of a society as we make every citizen count. In a census, all are equal, the rich or the poor, the young or the old, every minority enumerated for all to see. It is the candle in the darkness that shines a light on society and exposes our weaknesses and triumphs. But there are those who wish to extinguish this flame of impartiality for their own ends, they are those who wish to hide the inequalities of life, shun the impoverished and disenfranchised and maintain the bigotry that hurts us all.

When we are besieged by such evil forces, we cannot falter, and by cancelling the New Zealand census, you give these malcontents the fuel they need to further jeopardise the ability for all statistical agencies to fulfill their vital role. If you continue with this action you harm us all, and I plead of you, continue the census. Give the census a firm date and give the people of your nation a glimmer of light and show them that not even shattered earth will stop a statistician’s resolve. Those who survived may count themselves lucky, without your enumeration their efforts will be for nought.