Scripting market research surveys

The use of scripting to process market research questionnaires and surveys has grown in recent years. But, first, what is scripting? It means using a programming language to prepare a questionnaire, to process data or to produce tabulations. In other words, it means using software that is command-driven rather than using an interface where you use a mixture of a keyboard and a mouse (often called GUI or graphical user interface).

Scripting: Past, Present, Future

Scripting dates right back to the 1970s when tabulation software, such as Quantum, used a scripting language to specify variables and cross-tabulations. In those days and until maybe, 2010, many referred to this activity as specwriting (or, even, the more slang-like ‘speccing’). Indeed, some still use the term today. Scripting has grown now to include the specification of questionnaires, typically for online surveys and, in some cases, managing data and reporting. MRDCL’s developments for 2021 will take this one step further into the scripting of report automation to prepare reports in PowerPoint, for example. But, more of that later.

Scripting vs. Programming

You might say that I am pedantic, but I differentiate between scripting and programming. Scripting, in my view, is using a language within a software package. In contrast, programming uses an external, general-purpose language to achieve what you want, a programming language you could use to do anything. I am not keen on software that lets you ‘escape’ to a programming language when it runs out of steam. The reason is simple. It can be problematic in the medium to long term, particularly if you can use more than one language. It can mean that you have a ‘solution’ that no one knows how to update or, even, understand fully. I know some software uses this escape route to deal with situations where the software can’t do what you want, but I do not like this compromise. Why? If the programmer leaves, it might be, at best, very difficult to make the simplest modifications; and, we all know how market research surveys change.

When are scripting languages better than user interfaces?

As a company whose frontline product is MRDCL, a scripting language for tabulations, I am fully aware that MRDCL is not the right solution for everyone. So, when is a scripting language better than using an easier-to-use software with a friendly user interface? I would suggest these reasons and no others:

• A high volume of projects for analysis
• Complex requirements are needed
• Some large projects or projects with highly repetitive needs
• Some special or complex need
• Aiming to automate a process
• To embed MRDCL in another system as a ‘black box’
• Likely to fulfil one of the above conditions soon

Let’s look at each in turn.

When you have a high volume

If you have a high volume of work where someone or a team is spending most of their working time carrying out tasks where scripting is available, it may make sense to use a scripting language. Scripting languages typically offer shortcuts, and users can learn efficient ways to perform identical or similar tasks more quickly. In such cases, scripting can be the right solution.

When you have complex requirements

Generally speaking, scripting languages will have more power than programs that you drive through a friendly user interface. Of course, it’s not always true; however, there are only so many options that can be made available on forms, from selections or dropdown lists. If you have complex or, maybe, precise or fussy requirements, you may need a scripting language. It’s precisely why data processing agencies tend to use scripting languages so that they can serve their clients fully and meet their exact needs.

When you have large projects or projects with highly repetitive needs

You can process large projects or projects with repetitive needs in more straightforward software; you do not need a scripting language. The practicality that you face is that most straightforward software platforms do not handle repetitive needs well. Take, for example, big rating scales on a questionnaire that you need to repeat for four different products. Often, GUI software will only provide laborious or less effective ways to handle these repetitive tasks. In such cases, a scripting language can save time and make changes easier.

When you have some special need

Scripting languages tend to be more agile, as I have already discussed. If you have some special needs or work on a particular type of survey, a scripting language may simplify, semi-automate or automate part of a process. Tracking studies and multi-country surveys are types of survey where MRDCL proves especially efficient. However, if you only run one small tracking study per year, there are probably cheaper ways to manage the task.

When you want to/aim to automate a process

Scripting languages tend to be better at automating processes, though not always. Regardless of this need, some software platforms connect well to other systems, and some try to lock you into their system. This matter is, in practice, an entirely different issue. Yet, the design of scripting languages is usually to make automation (or semi-automation) more practical. Research and screening are, of course, necessary if you are looking to invest in a product to automate tasks.

When you want to embed MRDCL in another system as a ‘black box.’

MRDCL is an ideal engine to integrate into some other system. For example, where your system drives out the MRDCL script to do whatever you want it to do, MRDCL’s engine does all the hard work to calculate results and produce outputs. In this case, you are making the most of MRDCL’s power, but there is no need to edit or inspect the script usually.

When you are likely to fulfil one of the above conditions soon

If your needs are likely to change in the foreseeable future or expand your goals, scripting languages are less likely to put up barriers than GUI software. However, it is easier to assess where the obstacles for a GUI application are, whereas scripting languages may have more blurred boundaries making assessment much more difficult.

Thoughts about the staff you need

A major consideration is the staff you need to drive a product that uses a scripting language. You are likely to need more qualified staff for a scripting language, which may be more expensive. Further, to become proficient, the staff are likely to need more time to learn a new scripting language. In my experience, the payback for switching from GUI software to a scripting language is likely, in most cases, to take a year. The biggest benefit of a good scripting language is that the gains will keep increasing if you retain your best staff.

Hyper-productivity!

Scripting languages offer the chance of hyper-productivity – with the right staff. A GUI software product may mean that a highly productive worker can achieve, let’s say, one and half times the amount of work of someone who is averagely skilled or talented. Maybe, twice as much. With scripting, that ratio of productivity can leap to five times more productive than an average worker. Some scripters are fast, some are creative and highly skilled, and some are very accurate. Of course, those that are fast, creative and accurate are gold dust. Scripting does offer the chance of hyper-productivity; GUI generally doesn’t.

Getting the right staff

Getting the right staff to use a scripting language used to be difficult. It is becoming easier as more students around the world leave school or college with computing and logic skills. Good scripters do need a mix of skills – speed, accuracy and thoughtful work. Many scripters underestimate the time they take detecting and correcting errors as well as checking their work; accuracy is highly important as errors often impact others in the production chain.

Where does MRDCL fit?

MRDCL is predominantly a scripting language, although we have added features that move it towards a GUI product. However, its roots will not change; it will fundamentally be a scripting language aimed at those equipped to use its strengths in companies that fit the above criteria. Surrounding MRDCL are products that give you flexibility. You can use QPSMR, which is 100% compatible with MRDCL, for example. QPSMR offers a way to use a GUI product for simpler work while providing you with the option to switch to MRDCL if things get more complex as QPSMR and MRDCL use the same engine. You can also use Resolve, which allows you to convert an MRDCL project so that colleagues and non-experts can produce analysis using a easy-to-use interface.

Flexibility and automation

The immediate development goal for MRDCL is to build on its productivity focus by offering flexibility, connectivity to other products and automation tools. The latest release of MRDCL covers all these three goals – but there is more to come as we make the path from survey creation to reporting and providing data in dashboards to clients as easy as possible for not just simple projects but any project. Watch this space!