This is the fourth in a 5 part series of working with questionnaires and surveys managed using the Data Documentation Initiative XML standard. With DDI being an emerging technology, it is important that users are provided with best practises to ensure that they use the standard in a way that is logical, coherent and most importantly usable and reusable. This series of tutorials and discussions is aimed toward users who have some knowledge of DDI and would like to know how to effectively design markup existing and future questionnaires in DDI.

Part 4: What can I say?

One of the more useful aspects of electronic forms is they give us the ability to provide instant feedback to the user regarding their answers. HTML5 web forms especially are able to provide instant feedback to users on invalid data in number of ways. More importantly, electronic forms also allow us to perfom critical validation on data to ensure that we gather the most accurate data possible.

First off, lets look at the simple issue of finding what metadata is available in DDI 3.1 to provide validation. In DDI QuestionsItems contain the abstract idea of ResponseDomains, which in turn can be used to provide hints for systems to encourage certain responses from users, including Codes or Categories, various types of Numbers, Strings, Dates and Times, or Geographic data. Each of these explicitly imposes a specifc type of expected answer for a question, as well as additional restrictions:
For example, when asking for…

• age we can explicitly request integers over 0
• percentages we can define responses to be decimal numbers between 0 and 100
• location data, we can define the geographic coordinate system
• a unit code for a university course, we can specific a code scheme this must come from
• a post code as a string, we can specify a pattern the string must match

This is just a short list of possible examples for the use of ResponseDomains, and if there is an issue with restricting responses this is the first place those restrictions could apply. ResponseDomains can even specify top and bottom codes, if for example one wanted to top code ages censor ages above 65. With such a wealth of information available, the only issue becomes using this effectively to assist in response validation.

When looking at validation, web pages can act quite different to desktop applications. On a desktop application the programmer has a lot more control over what the user does and what data gets entered, however on the web, the client can effectively rewrite the webpage before submission making data validation much more important. What both paradigms have in common is that both will unfortunately interact with a user – the least secure, most buggy and most important part of any system.

Client-side / UI validation

When talking about client-side validation we are discussing ways to ensure that the user is effectively engaging with the system. By quickly and politely notifying the user of their mistakes, or gently preventing them from making mistakes we can try and prevent frustration, hopefully increasing response rates.

The main theme from my talk from the European DDI User Group in Utrecht last year, was around looking at how DDI can be transformed for use on the the web. With HTML 5 offering so many new features for data collection on the web, using DDI to leverage these new capabilities. Lets look at two examples from the slides for that talk.

 

In these examples the colours indicate how the metadata in DDI can be used to populate a webform. Although there may be debate about the use of some DDI data for certain parts of the HTML Form, this demonstrates how this data can immediately be used. To see this in action there is an example at sandbox.kidstrythisathome.com.

For legacy browsers that don’t support HTML5, this still isn’t a huge issue as these can ‘fake’ a lot of this functionality with Javascript. This can be accomplished either through the use of libraries to add HTML5 functionality to legacy browsers or designing transforms and proprietary Javascript to handle the user interface. The idea however, is to provide users with gentle hints to encourage them to provide the ‘right’ answers (ages aren’t words, and must be over 0) not to provide security about the data…

Server-side / data validation

That is the role of server or data validation, to ensure that the data that we have recieve is correct. As mentioned, on a webpage hints for the user are provided using javascript or through browser APIs. However, it is trivial for a user to bypass this on a client. Client-side validation should not be relied upon as the client can transmit back almost anything. The issue is that server-side validation is must less well defined: while HTML is a web known standard there is no ubiquitous langauge for server-side systems. However, the trade-off is that the programmer has complete control over the actions of the server.

So what should a programmer do to support good validation of data? The idea I approach in Ramona, was that each question was an object of a question in its own right. Furthermore, by using object inheritence in Python, the question class was sub-classed so that each ResponseDomain in DDI had its own class. This allowed each sub-class to define its own exceptions and errors, as well as inheriting those of the question super class.

For example, the question class defined the ‘validate’ method, that each sub-class would futher implement in diferent ways. Numeric questions would check their upper and lower bounds and throw exceptions if the response was outside this range, string questions would check the response against the pattern, etc.. and the server would pass the returned value to each class to validate in order.

In summary, DDI provides excellent support for questionnaire designs looking to control response from users and with this metadata provided in an accessible way providing application support for this an easy way for developers to promote DDI by demonstrating the real-world applications of DDI.

Next up… Questionnaire Design with DDI – Part 5: Can it be done? – A look back at the previous 3 tutorials and an indepth explanation of the realisation of a Ramona – A functional DDI Questionnaire tool.