For each paper if I’ve given an approximate amount of time needed to devote to the read it, and a suggested beverage for while reading.

GO TO Statement Considered harmful (A Case against the GO TO Statement)
E.W. Dijkstra, 1968 – (Morning tea & A cup of coffee)

This is probably the most recommended paper on this list, not just by me. Its no coincidence that it comes first, as not only was it the paper that has had the greatest impact on computer science, but it was released chronologically before the rest of these papers. Dijkstra efforts on structured programming were vital in making the rest of the papers on list list even possible. This paper is the demarcation point between unstructured and structured programming and put the first nail in the GOTO coffin.

A Relational Model of Data for Large Shared Data Banks
E.F. Codd, 1970 – (2 or 3 evenings & a bottle of red wine)

Codds paper introduced the world to generic relational databases. While we take relational databases for granted, this paper convinced the computing community of the benefits of relational algebra and structured storage for data. While this paper will present nothing new to a seasoned database developer, it presents an concise coverage of the advantage of relational databases and is a counter-point to the current NoSQL/NoRel trend.

On the criteria to be used in decomposing systems into modules
D.L Purnas, 1972 – (An evening & 2-3 glasses of scotch)

Purnas presents the ideas of modulatisation in programming. Again, modern programmers can probably take for granted the idea of a standard library in a programming language. But this paper dictates why these are vital and brings up ideas of how to balance modulatiry and reuse versus too much modularity and being to complex to reuse.

On the role of scientific thought.
E.W. Dijkstra, 1974 – (A lunch break & a bottle of water)

In this paper Dijkstra brings up the idea of the separation of concerns. Where as programmers we view each part of a program as its own discrete concept in isolation of (but not ignorant) of the rest of the program. This paper essentially defines the idea of the separation of presentation and content that we take for granted in web development.

Applications Programming in Smalltalk-80(TM): How to use Model-View-Controller (MVC)
S. Burbeck, 1974 – (An afternoon tea & a cup of earl gray)

This last paper is a technical and practical discussion of work introducing the “model-view-controller” paradigm to programming. While interesting from a historical perspective as an extension of the previous paper by Dijkstra, it also serves as a good introduction and arguement for separation of concerns in a practical way.

Lastly, there is a list of important publications in computer science available on Wikipedia that has a greater range of content that is slightly harder to get access to, but for someone with journal access and specific interests, this is an excellent starting point for discussion.

My most recent read was E.F. Codd’s “A relational model for large shared data banks“. Reading Codds paper, you find a paper that succinctly lays out the advantages of relational design, in easy to read language and put forwards a strong argument for their use. In 10 pages Codd covers relational databases compared to flat files, introduces relational algebra and set theory, discusses elementary issues in scalability and covers the basics of redundancy and consistency.

But this should be no surprise, because this paper that kicked of the relational database movement. In his paper, Edgar Codd (as in Boyce-Codd Normal Form) convinced the computer science world that flat files weren’t good enough for large scale applications and that a better solution was needed.

Looking back as a relatively recent graduate of computer science I have to ask – why have I not read this paper before? There was nothing that Codd said that I hadn’t read in lecture notes based on more recent work, and it was no more or less easy to read that anything else I’d read. But this was not just any article justifying relational databases, this was the article.

Sadly, this was all too common, and most of my studies were based on recent works, rather than examining the classics in computer science. And this leads me to believe that this is part of the reason we see so much repetition in computer science. Not because we are reexamining the well, but we are reinventing the wheel. At an undergraduate level I never saw the drive  to explore early computer science theory (partly my own fault, but also of my lecturers for pointing them out).I don’t think this was due to malice or lack of access, as many of these papers are available for free or through a university library. Mostly, because with the fast growth in computer science, what is new becomes old very quickly,and with so much information being generated it takes effort just to keep up. But based on my reading of Codd, I can say that there is no good reason not to look back When trying to rationalise the use of a relational database, who better to convince a student than the man who literally convinced the world.

So, with this new found access I’ve begun rereading these ‘forgotten’ papers, because there is still so much to learn from them. In Codds case, I’ve learned that there is no better source of information than its originator, and will encourage others to read his work as well. In hindsight his work seems obvious, its historical context and ability to share this idea sets it apart from every other database text I’ve read. If you get a chance to read this work, which is available freely thanks to UPenn, I strongly recommend it.

[edit] A relational model for large shared data banks

Flash forward to now – I’ve long since left that job, now working as an Information Analyst for the Australian Government. Most of my work in business analysis and theoretical computer science. I barely even administer my own home network and have forgotten most of what I know about networking.

So why am I not renewing my CCNA?

Because it is intellectually dishonest. At the time, when I got certified, I got it as a proof of my understanding or at least ability to understand. While my career was still in its infacy, this helped demonstrate my capability. Now however, I have experience as a programmer and analyst, but no longer working with networking.

Certification is in essence two things: a resume substitute for experience and proof of ability. If I was to renew my certification (which would require quite a bit of relearning) it would only be a disadvantage, for me and for every other person with a CCNA accreditation. If in the future I was hired on the basis of my accreditation I would soon be found out as someone who is now relatively unskilled with networks, which would only make me look bad. Worse though is that by me, or anyone, who holds an accreditation without appropriate experience or skill only make everyone else who holds that certification look bad by comparison, by the reducing the surety of the accreditation as an indicator of that skill.

As such, holding any certificate without the appropriate skill to back it up is both not in my own self-interest and morally bankrupt.

In all three questions, respondents are asked about their origins, and are given the option to select from a list of common responses or provide a write in response. The easiest way to manage this is through the use of a DDI <MultipleQuestionItem>. A <MultipleQuestionItem> is a way to capture a complex question that asks two or more separate questions that are highly linked.

In the above examples we can split the questions into two, as illustrated in the generic answer below:

<MultipleQuestionItem>
    <SubQuestions>
        <QuestionItem>
            <QuestionText>
                <LiteralText>
                    What is your ancestral origin?
                </LiteralText>
            </QuestionText>
            <CodeDomain>
                <!-- This CodeDomain would include a reference to the list of countries or races -->
            </CodeDomain>
        </QuestionItem>
        <QuestionItem>
            <QuestionText>
                <LiteralText>
                    Please Specify:
                </LiteralText>
            </QuestionText>
            <TextDomain/>
        </QuestionItem>
    <SubQuestions>
</MultipleQuestionItem>

Here we have been able to split the question, while still managing it in a single item. This is needed as without each other, each subquestion is incomplete. This is not a new concept, and is quite an obvious solution to many people who have tried to solve this issue.

However, there is still the problem that this metadata doesn’t contain the restriction that a respondent should only be able to enter a free text option if the “other” option is selected. While there have been a number of published and attempted solutions, none have been satisfactory. Spliting the question outside of a MultipleQuestionItem and using IfThenElse clauses complicates the structure, and leaving this out makes designing self-interviewed computer systems difficult to manage directly from the metadata.

A possible solution, that resolves both of these issues is through the use of the <SubQuestionSequence>. This is illustrated in the DDI Fragment below:

<MultipleQuestionItem>
    <SubQuestions>
        <QuestionItem>
            <!-- Ancestral origin QuestionItem -->
        </QuestionItem>
        <QuestionItem>
            <!-- Please Specify QuestionItem -->
        </QuestionItem>
    <SubQuestions>
    <SubQuestionSequence>
        <ItemSequenceType>Other</ItemSequenceType>
        <AlternateSequenceType formalLanguage="Name Of Language Here" >
            <!-- Proprietary command to control logic -->
        </AlternateSequenceType>
    </SubQuestionSequence>
</MultipleQuestionItem>

In this we have used the SubQuestionSequence to hold the logic used to indicate when the “Other” field should be allowable. This field is used to control the specific sequence that the SubQuestions are shown, and in this sense we are controling this ordering, just to specify when a member is not shown – an excusable use of the field. This choice can be further rationalised, as an unfamiliar agent, for example when moving to a new piece of software, can still interpret the bulk of the metadata, however when presenting the above question would allow a respondent to fill in both sections. But this is no different to how a respondent of a paper-based survey may answer, so it is no great loss of granularity.

How any given agency may choose to populate the commands contained in the AlternateSequenceType will be an individual choice, and a standard way of expressing this may be needed, but this should help other groups more easy solve this problem by indicating where the solution can go and reducing the problem size.

In the next day or two I will be putting a more solid example up into the DDI Examples Repository for people to work with. As always critiques of these ideas and examples are welcome.

At the moment there are two ways that this can be done: I can either make volunteers permanent contributors to the repository so they can share files directly, or people are free to raise tasks and files can be uploaded by myself or another contributor.

I have also split the repository at the moment into three sections:

1. Simple Examples: used to illustrate a simple concept or use case – such as a Question, Concept or Study – but not ideal for actual implementation.
2. Complex Examples: to demonstrate more complex ideas, such as referencing, packaging or schemes.
3. Practical Examples: highly complex examples that replicate an actual problem or real world scenario completely.

At the moment, there are only two files up on the site – the Australian and New Zealand Standard Industry Classification and the Australian Drugs of Concern Classification. Both of these were generated using Virgil-UI and have had some minor tweaks applied to they are DDI3.1 valid.

With time, I hope this can be a comprehensive resource allowing users easy access to DDI example to lower the barrier to entry to the standard.

It was pointed out that I made a small error when dealing with <Code> elements in DDI and accidentally gave them @id attributes, and it was noted that this should be an easy fix. Unfortunately, due to my missing this very early on in the development of Virgil the underlying model relies on Codes having ids to be able to easily make connections between the hierarchical user interface, the <Code>s and the <Category>s that give them meaning.

What this means is that both the DDI coming out of Virgil is invalid, and any valid DDI would not actually be able to be read by Virgil. Essentially, the Virgil model for handling DDI is broken and needs to be almost entirely rewritten and this might take quite a while.

Unfortunately, at this stage rewriting also means re-examining a lot of the initial ideas about what Virgil should be and has highlighted some interesting questions about the DDI model and DDI software, such as:

1. Is abstracting the DDI model away from a user a good approach to software design? Yes.
   This was the crux of my talk at EDDI, and I still feel that abstracting the DDI model away from day-to-day users is necessary. The DDI model is complex and covers a wide range of tasks. I believe that designing software that helps users relate the model to specific tasks they are trying to do is a key to getting people to use DDI and think about how they can make their metadata support themselves and those around them.
2. Is DDI a standard that is suitable to use for day to day management of information? Probably.
   In practice, the DDI standard needs to be able to be passed between software if it is to move from an archival standard to a practical statistical metadata standard. One of the things I wanted to achieve with Virgil, was a tool that not only produced DDI, but could also consume it from other sources. In the simplest case this to me meant being able to take a DDI file, and edit the contents of part of it, leaving the rest untouched, and in a lot of cases this is possible with DDI. However, since having to rethink how to manage classifications using DDI, I have realised that there are some objects that are not captured well within DDI and unfortunately classifications are one such example.
3. Is the DDI model for managing codelists and classifications good enough? Sadly not.
   One of the reasons I relied so heavily on the invalid <Code> @ids was that I needed a hook to tie codes and categories together and without this it becomes very difficult to manage what a ‘classification’ is in DDI. Furthermore, classifications don’t exist in DDI per se, but are a rather loose agreement that if you combine <CodeScheme>s and <CategoryScheme>s you get a good approximation. However, this falls apart when we try to document the classification itself.
   For example, where do you store the name of a whole classification? There are three viable places (excuse the XPath) – as a //CodeScheme/Label (being the label of the hierarchy), as a //CategoryScheme/Label (being the label of the collection of classifying categories) or as a //LogicalProduct/Label (the label of the immediate parent that contains both the hierarchies and the categories).
   However, each of these approaches has inherent issues, as neither of these are the documented way to manage this information, and if 3 different agencies approached the problem in different ways, then their metadata becomes incomparable. This needs to be discussed further, as it will become a bigger issue as more tools start to try and manage such an important, and conceptually early in the lifecycle piece of metadata.

It should be noted that these issues don’t excuse overlooking the actual standard leading to this predicament. However, given the chance to re-examine how to correct the problem in Virgil, also gives me a chance to examine some of the issues I came across while trying to maintain classifications within DDI. Over the coming month or so while I am going to continue writing up some of the issues I identified with classifications within DDI3.1, how to work around these in the short term, and look at ways to correct the problem in future versions of the standard.

Lastly, in the short-term there will be an update to correct the Code/id problem in the CSV to DDI conversion, so the original use case of being able to mine legacy systems to produce valid DDI will still be filled.

Thanks again to everyone at EDDI for their input and company.

By all accounts, the European DDI Users Group meetings were a great success. Along with seeing all the work people have done of the last year, we were able to sit and discuss and debate for several days and have a solid plan for future work.

While I was only at the Developers meetings, we covered improvements to the website, new ways of managing large DDI instances in relational and non-relational databases, examined new (and forgotten) ways to design software, debated the best ways to handle automated ID creation, listened to the results of the semantic DDI workshops, learned about the DDI Agency Registry, debated reducing or removing namespaces from DDI, raised the possibility of a shared DDI Blog/News aggregator and started the creation of not one, but two major additions to the DDI community – a new transport element nicknamed “The DDI Bucket” and started laying the groundwork for a DDI RESTful web interface standard.

And that was in just 3 days! And I am still eagerly awaiting to see how the “Data Without Borders” and “Longitudinal DDI” workshops went.

The week was made even more productive by the use of Google Docs to create a single, living recollection of the event. Watching everyone type up their notes in real time was great. Over the next few week I (and hopefully the rest of the DDI Developers community) will continue to clean up our collaborative notes and look forward to presenting information and recommendations to the whole DDI community in the new year.

We also discussed upcoming meetings for the DDI Developers group and 3 possibilities were raised, at IASSIST in June, RC33 in July and EDDI next December. While events will most likely go on at all of these events, I strongly encourage those who can come to RC33 to be held in Sydney next July to speak up or at the least contact me in private. There is a wealth of talent in Australia and New Zealand who are well worth getting in contact with and with a large enough group of DDI members in Australia I think a “DDI Developers Down-under” would be well attended and well worth the trip.

So with that in mind thankyou to everyone in the DDI Community for a great week – and especially to Olof Olsson of SND for kindly offering me a place to stay during the week. It was a fantastic week, and served to remind me how if you work hard you can contribute to a community, being called upon to answer questions during the meetings (and once during the question time of someone else’s talk!) was especially flattering. This has truly re-invigorated my love of metadata (I spent the better part of my evenings in Rome madly writing ideas for tutorials and examples I foolishly volunteered for during the meetings)

So, with that I wish the entire DDI Community a Merry Christmas, Happy Holidays and Happy New Year and look forward to seeing everyone again in the new year, be it in Washington for IASSIST 2012, Sydney for RC33 or wonderful Bergen for EDDI 2012!!!

Arrivederci

https://lh6.googleusercontent.com/-Bt7M3EmQOVo/TuY2PWCS8hI/AAAAAAAAEOc/nmKN-M6tK-M/s512/IMG_20111211_184959.jpg

However, it isn’t and I wont.

So throughout EDDI, I have been compiling a list of gripes and grumbles (as well as positive remarks and suggestions) regarding the website. In the new year I will be sending out a short survey to DDI Users looking at how people use the website, their issues (both positive and negative) and how they think the DDI Website should look in the future.

One main issue that will definitely be addressed as a part of this exercise is the lack of positive examples of DDI available on the web. The reason for this is that the survey itself, and all its metadata will be made available for people to download and study. This will not be an easy job, but I look forward to contacting researchers and developers across the DDI community to help piece this together and make improvements for everyone.

Sing a song of software,

To test this out I put together a little tune, that covers the past 24 years of stock information from Microsoft (Piano), Apple(Clarinet) and Google(Xylophone). The pitch is proportional to the price of the stock with low tones being low prices and high be high. While the volume of each instrument is proportional to the volume of sales over the period, so when you hear a quiet sound that is a low volume day, while loud note is a period of higher trading volume.

There are two versions of the music available:

A shorter 2 minute, up-tempo version using weekly stock prices: OGG, Midi – This one is short and to the point, but some of the nuances, like big daily trade spikes are missed.

A longer, 17 minute, version using daily prices: OGG, Midi – This one is a little monotonous at the start, but you can hear Apple come from a tiny instrument in the background to a larger force much better. It also lets you hear some off Apples big trading days.

Bubbles full of lies;

A few things to listen out for:

• Early on, listen for Microsoft’s speedy accent during the 2000′s tech boom, and an even quicker decline. (About 1:00 in on the quicker version)
• Apple, has for a long time a consistently low trade volume, however occasionally you will hear loud piano strikes starting from the early 2000′s. (About x minutes in.) These are peaks of stock sale, probably around MacWorld and iPod/Phone/Pad announcements.
• After about 2005, you can hear Apple and Google slowly rise in volume and stock price, while Microsoft remains in a consistent range throughout the same period. (After 2:00 in the short version)

4 and 20 years of stocks,

The data that all of this was pulled from was the historical stock prices data sets available on Google Finance. Why 24 years worth – because it fit with the theme of the nursery rhyme I was trying to mimic. Its pretty touch and go as to what data you can download from Google Finance, but to be fair, from my understanding this is an issue with the exchanges rather than with Google.

Audio-lised with Py(thon).

So the nitty gritty on how it works:

Its a python script that loops through a set of files of output data from Google Finance and using midiutil creates a Midi file. Each day (or weeks) datapoint is weighted so the values remain within a specific range for a specific instrument and the volumes are adjusted so that each instrument can be detected. Without either of these it really is quite a mish-mash of sound.

This output Midi file is then run through Timidity++ to create an Ogg/Vorbis file. Converting to Ogg is only necessary for consistency, but both the Midi and the Ogg are available.

Future work and ideas

Well the goal is to be able to use a technique like this to listen to large multi-variate datasets, that have either a time dimension, or a continuous dependent variable (heights, weights, etc…). As long as one dimension has values that are relatively evenly and closely distributed with few overlaps and a wide enough spread it should be possible to ‘graph’ probably any dataset meaningfully as audio.

Not everyone will find programming fun or interesting, however at more than one time people will come up against a problem that computers were made to do. Contrary to popular believe computers are dumb – at least in the sense that they can only do what they are told. What they can do, is they can do these dumb things very, very quickly. So much so, that they can fool you into believing they are smart – more than smart, magic even. In fact, if you are even a mediocre programmer people can become convinced you are a magician.

So why should you learn to program?

Mostly, because your time is valuable. Not to me, but to you. Unlike a computer your time is finite, and if you can make a machine that can give you more time to do something, isn’t that in your interest? Even if it isn’t in work, if its just sorting your taxes, or writing a script to check your email for you, there are plenty of small, repetative tasks that you probably do that a machine can do quicker. If you enjoy doing repetitive tasks, then there isn’t much I can do for you. But if you want to spend more time understanding why you do these things, then read on…

Where do you begin?

Well, I think, there are 3 programs every one must be able to write. Because if you can write these 3 programs and adapt them to your needs, you can do most big, boring tasks that will come your way.

There are 3 programs you need to learn to start to become a programmer and as I explain them, I’ll show you a brief example to edit and play with and ultimately understand what is going on. These examples are written in Python, a free programming language with a very user friendly syntax.

Hello, World!

“Hello world” is traditionally the first program many new programmers will write. It is simple, when the program is run the computer prints “Hello, World!”. In essence, this simple program introduces programming syntax and demonstrates how to display text to a user.

print "Hello, World!"

There isn’t much to this, but its a starting point. It teaches you some basic syntax and with a lot of languages understanding syntax is important – computers don’t speak English and to make them useful you need to learn to talk to them, more than the other way around.

Simple user interaction

The second is a simple string manipulator. This goal is to create a simple, persistent user interface, with some error checking that fulfills a task. Here we see an example that does actions on a given string based on a command given to it.

while True:
        input = raw_input("> ")
        try:
                cmd,text = input.split(":",1)
                if cmd == "uc":
                        print text.upper()
                elif cmd == "lc":
                        print text.lower()
                elif cmd == "rev":
                        text.reverse()
                        print text
                elif cmd == "quit":
                        print "Bye"
                        break
                else:
                        print "Command not recognised"
        except:
                print "Syntax error: enter a command, a colon (:), then a string"

Firstly, we start the loop and set it to never stop looping. As long as the user wants to play with strings this program will keep going. Next we ask for some input from the user.
Now things get a little more complex, first we try and split the string around a colon, into a command and the text. If the user doesn’t enter a colon, we throw an error, and give them some help text (after the line that says “except:”.
If they do enter a command and text, we set the text to upper case, lower case or reverse it. If they tell us to “quit:” we quit by breaking out of the loop (the break command) or we tell them we didn’t recognise the command.

Its not perfect, but it gives us an understanding of errors, handling user input and basic user interaction – not bad for 18 lines of code.

File manipulation

The last and most important program is a simple file manipulation tool. Again, what the tool does to the file is irrelevant- it might merge files, look for spelling errors, count lines, anything. Perhaps, we are looking for entries in a diary that start with numbers (like dates) in a large file, and only want to view these.

file = open('test')
for line in file:
    if line[0].isdigit():
        print line
file.close()

Here we open the file, and then line by line we search through it. When we find a line whose first character is a digit we print the line (line[0] essentially means the 0th character from the start – its complicated but its how almost everyone deals with subscripting in lists). Lastly, we clean everything up by closing the file.

Again, by no means the best implementation, but easy enough to read and alter. This time in 5 lines we have a simple script that could help use find our tax information, search our diary, lookup phone numbers, or anything like this.

I’ve done this, what now?

Do whatever task it is that needs doing. Odds are having read these short code snippets you can get an idea of what can be done. Its just a matter of getting out and doing it. You may not be the next Bill Gates/Mark Zuckerburg/whoever, but if you learn to paint a wall you won’t be the next Da Vinci either. Learn how to do what you need to do, and keep plugging away. Programming is about pulling together pieces of logic, to help us do simple tasks easily and reproducibly. So think lazy, and find all the tasks that you can automate and get something else to do them for you.