Questionnaire design with DDI – Part 1: Will this survey ever end?
This is the first in a 5 part series of working with questionnaires and surveys managed using the Data Documentation Initiative XML standard. With DDI being an emerging technology, it is important that users are provided with best practises to ensure that they use the standard in a way that is logical, coherent and most importantly usable and reusable. This series of tutorials and discussions is aimed toward users who have some knowledge of DDI and would like to know how to effectively design markup existing and future questionnaires in DDI.
Part 1: Will this survey ever end?
This is a classic question that survey takers often ask, and not usually that politely. When a survey will end is an important question for users and designers alike, as the time users have to spend filling out a form can impact their likelihood of finishing or even starting a survey. So determining if a survey will end is of vital importance for users of DDI.
Unfortunately, the answer, based on simple analysis is that you can never tell. DDI uses a unique methodology of using complex referencing between sequencing objects to build up the structure of a questionnaire in a way that is highly reusable. Take for example the following pseudo-survey:
Survey: All about You!
Part 1: Feelings
Q1: Are you feeling happy today?
Q2: How do warm summer days make you feel?
Part 2: Favourites
Q3: What is your favourite icde-cream?
Q4: What is your animal?
Now, restructuring this in a way more analgous with DDI gives:
Sequence id="Part 1" title="Feelings"
question reference: Q1: Are you feeling happy today?
question reference: Q2: How do warm summer days make you feel?
Sequence id="Part 2" title="Favourites"
question reference: Q3: What is your favourite ice-cream?
question reference: Q4: What is your animal?
Instrument id="survey" title="All about You!"
sequence reference: Part 1
sequence reference: Part 2
The important thing to notice is that both Part 1 and 2 and the main survey are all the same structure: a DDI Sequence. In DDI, to increase reusability sequences can be included by reference within each other. However, the issue comes when a designer (either a person or piece of software) fails to account for this sending the survey taker into an endless loop. For example, looking at a different survey strucutre like DDI gives:
Sequence id="Verse 1"
question reference: Q2n-1: Is this the song that never ends?
sequence reference: Verse 2
Sequence id="Verse 2"
question reference: Q2n: Does it go on and on my friends?
sequence reference: Verse 1
# Apologies to Lamb Chop
An even shorter (but much less likely) example would be:
Sequence id="infinity"
question reference: Qn: Do you believe how vastly, hugely, mind- bogglingly big Inifinity is?
sequence reference: infinity
# Apologies to Douglas Adams
What these two examples show is how it is possible to send a user through the same sequence twice, potentially leading to an endless loop of questioning. With DDI providing appropriate mechanisms for Looping over questions, it is quite likely that this kind of structure will always be the result of a mistake. However, what it does highlight is how the flexibility DDI provides to allow users to reuse metadata when used inappropriately can cause issues.
Fortunately, as part of the research in developing a DDI web application for data collection, it was neccessary to create a module that was able determine the implied structure and possible ending of DDI Questionnaire. A web-service for this module (aka. Shari) is available online at http://sandbox.kidstrythisathome.com/dditools/sheri.
One of the notable features of Shari, is that is will create a hierarchical non-DDI based XML serialisation of a given questionnaire (the reasons for this will be covered in Part 2 of this series “Where am I?”). Along with this it will check that no two sequences have references to the same sequence to confirm that the questionnaire will halt. However, the ‘halting’ of the survey is however based on the provision that any DDI Loops within the survey are also able to end, but this is a true ‘halting problem’.
By throwing an error when ever a sequence is referenced twice, some may believe that this will lead to a perceived issue in which two sequences in different branches could link to a third sequence without causing an endless loop being rejected by the system. However, it is possible to rewrite any instrument that relies on using multiple references to a sequence to ensure convergence of a survey in a way that eliminates duplicated references. For example:
Instrument id="Main"
sequence reference: Seq 1
Sequence id="Seq 1"
question reference: Q1: Do you like A or B?
if Q1 = A then goto sequence 2a else goto 3b
Sequence id="Seq 2a"
question reference: Q2a: Why do you hate B?
sequence reference: Seq 3
Sequence id="Seq 2b"
question reference: Q2b: Why do you hate A?
sequence reference: Seq 3
Sequence id="Seq 3"
question reference: Q3: Wouldn't it be better if every one got along?
In this minimal example, it is trivial to see that this survey will always end. What is important to note is that this can be rewritten as:
Instrument id="Main"
sequence reference: Seq 1
sequence reference: Seq 3
Sequence id="Seq 1"
question reference: Q1: Do you like A or B?
if Q1 = A then goto sequence 2a else goto 3b
Sequence id="Seq 2a"
question reference: Q2a: Why do you hate B?
Sequence id="Seq 2b"
question reference: Q2b: Why do you hate A?
Sequence id="Seq 3"
question reference: Q3: Wouldn't it be better if every one got along?
As the users steps through the branch, after either of the sequences 2a or b end, the survey steps ‘back’ from the inner sequence to Seq 1, not finding a following sibling for the If branch it steps back again to the parent instrument and finds Seq 1 has a following sibling and then steps into Seq 3.What should be taken away from this example, is that is is important before finalising a DDI questionnaire to understand the implied structure of the instrument and refactor it to ensure that it is minimal and logically correct as well as having the structure required by the survey designer.
In conclusion, it should be a little clearer about how to tame the flexibility that DDI allows when creating questionnaires and how to create logically correct survey instruments.
Next up… Questionnaire Design with DDI – Part 2: Where am I?- Examining why DDI has issues with non-predictability of movement through an instrument, and how to work around this.
You’re a few weeks late with this one sport…
Look at it like this, if I put it up earlier you could have had to compete with more people
But, of serious note is that it has become apparent how difficult DDI can be for some people to work with, and work with correctly. So I’m going to start putting articles like these out there to help people work with DDI better and quicker.
Hi Sam, I enjoyed your post.
With the new parameter bindings in DDI 3.2 [1], detecting user errors in survey flow creation can also be done at the point of branching by evaluation of the universe and conditional evaluation. This checking can also be done in DDI 3.1 but is more difficult since the entire instrument graph must be traversed to be validated since there was no explicit scoping. In 3.2 this validation can be done using any control construct as a starting root, which should greatly increase their reusability. This parameter flow based analysis would be more accurate than only checking for reuse of control construct references, which is often valid and necessary.
A modification of your method to do simple cycle detection in the instrument graph might be more appropriate than reference counting. If you think of the instrument as a directed graph, you can quickly find any existing cycles. Your final example above would pass this examination since it is a directed acyclic graph. As you mention, this does not account for users who never terminate their loop/while/until conditionals.
Cheers! Dan
[1] http://mantis.ddialliance.org/view.php?id=366
I should sit down and go through the DDI 3.2 spec as much as I have with 3.1, to get my head around what I keep getting told is the latest and greatest. But it would have to be a big change to the structure of DDI instruments to remove the above issue. If you are including any ControlConstruct by reference there is a risk of get trapped in a cycle. This isn’t meant to be an argument against DDI, but just a concern with how the structure of instrument is currently created. However, the current structure provides great reusability and as long as users are educated (which is the point of these tutorials) it shouldn’t be an issue.
With regards to the cycle detection, thats a bit of a pragmatic argument. My method could certainly be better structured to only find potential cycles, however then it wouldn’t address the issue of making a user go through the same questions again. This then becomes a debate less about graph theory and more about survey design, trying determine what is or isn’t appropriate to serve to users through an instrument. Personally, I am of the belief that it is a problem to have users fill out the same questions twice, so I built Sheri around that assumption. If other people have other ideas, if they are aware of the issues this creates then thats fine too.
However, the tool will persevere and continue making the structure, it just won’t expand any repeated elements. So if the user knew about the repeats and was happy with them the tool can at least validate the rest of the object. That said, I am looking at improving the tool, and perhaps drawing the distinction between repeated items and loops is a better idea.
Hi Sam,
You can look in 3.2 but you won’t find the piece Dan is refering to yet as it hasn’t been entered into the development schema yet. I’m assuming you have gotten access to the development site
I also assume that bugs and discussion of the parameters of these bugs will find their way into the bug list. You’re doing great work here and I’ll keep reading.
Wendy
Yeah I have access to the development site, but I haven’t really been viewing it as much as I should have. As of yet I don’t think anything I’ve described is a bug, more quirks or commonly misunderstood ideas that need a bit more rigor when put into practice by users.
Glad to know I’ve not just projecting all this stuff out into thin air!
[...] which may or may not be a security issue depending on the form. Alternatively, as discussed in the part one of these tutorials, restructuring Instruments so no structure is referenced more than once, making it easier to [...]
By looking at the post we can say that how much ur post has caught our eyes…. keep doing the great job