Producing Cross-Tabulations in MRDCL
A guide explaining why, how and when to use MRDCL
Cross-tabulations, often called crosstabs or tables, play a prominent part in analysing market research data. Cross tabulations offer a convenient way to compare answers from different types of survey respondents. For example, you may wish to compare the opinions of each age group or each region. You may want to see if there are differences between answers given by regular and occasional users of a product or service. We explain how to use and cross-tabulations in our Guide to Market Research Tables.
MRDCL is a software product with its own scripting language. The primary purpose of MRDCL is to make the processing and analysis of survey data as efficient as possible. The MRDCL language is full of features that make complex requirements possible and make repetitive requirements as easy as possible to specify and generate. Scripting software needs skilled staff to drive it, which comes with significant advantages in the right hands. Productivity gains of 50% are typical and, in some cases, as much as 80-90%. The MRDCL scripting language also means that you can satisfy any researcher or client need.
MRDCL is not suitable for every company. Like any software product, it has its strengths and weaknesses, making it ideal for some companies and unsuitable for other companies.
MRDCL comes in two parts, MRDCL Classic and MRDCL Central.
MRDCL Classic is the original program that market research agencies have been using for over 25 years. MRDCL Classic is the main engine of MRDCL that reads users’ instructions and produces the required outputs, mainly cross-tabulations.
MRDCL Central is the new addition to the MRDCL suite. MRDCL Central enhances some of the features already present in MRDCL Classic and offers an increasing range of tools to take MRDCL beyond tabulations and facilitate automation. For example, to enhance MRDCL Classic, there is a colour-coded editor. To improve productivity, there are tools to import data efficiently and to automate tables, reports and other outputs.
Having decided to use a scripting language for producing cross-tabulations, it may not mean that you should use MRDCL for every project. A project may be quite simple and not require the expertise of a data processing expert. In such cases, you may choose to use QPSMR to produce your analysis. QPSMR is fully compatible with MRDCL and uses the same engine to run your cross-tabulations. It also has tools for data collection and an add-on CATI module.
As QPSMR uses the same engine as MRDCL, you can instantly open your project in MRDCL and make use of the additional features in MRDCL, such as reporting automation if you wish.
QPSMR is available from several dealers. However, we recommend that, if QPSMR would benefit you, you should buy it from our approved dealer, MRDC Software offers a 50% discount on a QPSMR licence if you have already purchased an MRDCL licence. This approach gives you even greater freedom in how you process projects without losing the power of MRDCL.
MRDCL offers a new concept for market researchers to automate their projects and, in turn, their insights. Automation is not new to market research. Some of the more basic data analysis tools already have automation tools in place. However, MRDCL will be the first scripting language dedicated to market research to allow automation from start to finish for your projects. Although we have not completed all the developments yet, the MRDCL scripting language will allow you to set up a series of processes. For example:
MRDCL is the only scripting language meeting modern market research demands.
Market research has changed substantially over the last ten years. Like MRDCL, all the other scripting languages date back much further than ten years. What has changed is that market research has changed in terms of deliverables. The deliverables are more varied, meaning that you need a more flexible engine to interact with the software you need to meet your client’s needs.
Rather than have one process for your surveys, you need to be flexible, which means that you need to automate processes where possible. MRDCL has changed from being a scripted cross-tabulation tool to a scripted research processes automation tool (read more).
For the highest productivity, the ability to produce more or less any cross-tabulations and efficiency, you need a scripting language or, possibly, a hybrid software product. In this section, we compare the four major scripting languages. Let’s cover these in order of age (to the best of our knowledge).
Quantum is undoubtedly a first-class scripted tabulation software product. Since SPSS acquired Quantum in the late 1990s, there has been almost little or no development. The fact that it still survives in some companies is testament to its completeness and robustness. Later, IBM acquired SPSS and more recently Unicom took ownership of Quantum. In 2018, Unicom announced the release of an unimproved Windows version of Quantum, having worked solely under DOS (sometimes called Command Prompt). Despite this announcement and promises of more developments, as of February 2021, the one-page statement remains unchanged with no further news. Quantum was arguably the greatest cross-tabulation software in the history of market research. However, the lack of development in over 20 years means that Quantum falls short on functionality, automation tools, software integration and modern needs.
Uncle appears on this list due to its longevity in the market. The product is scarcely used outside North America and has almost no information on its website. New releases seem to be a rarity, but it seems to have its fans.
Merlin and our own MRDCL come from the same origins and still share much common syntax. Merlinco, the developers of Merlin, has focused its development on new minor features. We find many of these redundant as they are focused on text-based tables. In contrast, the majority of the market wants tables in Excel or CSV format. However, Merlin remains a good software product, although its developments have been negligible in recent years with an almost totally inactive website.
Finally, we come to our scripted tabulation software, MRDCL. MRDCL has had a series of clear goals, which continue to evolve. These goals and direction are what make MRDCL unique. Sometime shortly after 2000, we realised that MRDCL did more or less everything anyone wanted to do when producing cross-tabulations. That’s not to say we haven’t responded positively to some user requests over the last 20 years, but our focus has been clear. Below, we cover the focus of our developments in recent years. Further, we explain our commitment to taking MRDCL to a new generation of data processing software.
When considering our development plans for MRDCL ten years ago, the first issue we identified was the increasing costs of staff and data analysis experts around the world. For that reason, we focused our development on making users of MRDCL more productive. We added features and tools that made data analysts more productive, thus saving our customers money. Examples of where users can improve productivity are:
The second series of development has been to improve automation. With market research projects having more delivery paths, it has become increasingly important to offer automation. We have not completed all these developments yet, but MRDCL can already automatically import data to a framework, run analysis and deliver multiple outputs as one continuous process. We have already made MRDCL an excellent product to integrate into your other software and processes and it will continue to improve this area. We believe connectivity between software products will increase in importance as time goes on.
The third development in progress in 2022 is for MRDCL to have a new module to script charts and reports. The production of one or many reports will form a new part of the automation tools within MRDCL.
Switching to MRDCL is not a decision to take lightly. Although we offer free induction training, we have over 60 detailed videos covering everything from ‘getting started’ to more advanced concepts. Anecdotally, clients have found these invaluable.
There are many considerations when wishing to switch to any software product. We can help you to move to MRDCL in an effective way with many ready-made solutions to the challenges you will face.
The key considerations are:
Let’s look at each of these issues.
If the team has experience using a scripting language or a programming language, the time to learn MRDCL will be significantly less. Good knowledge of market research and data will make a difference for those unfamiliar with a scripting language. The most practical ways to switch to MRDCL will be dependent on previous experience.
Regardless of previous experience, knowing the best way to handle tasks in MRDCL can take time. Progress often unfolds in steps – from learning the basics to becoming familiar and becoming highly efficient. Everyone will progress at different speeds; we have a lot of experience putting together recommended learning programmes. We have some sample training programmes available for different types of user.
For some companies, transferring projects to MRDCL can be a time-consuming part of the process and can be a powerful reason not to change. If most of your projects are ad hoc projects, this is rarely a problem. However, if, for example, there are several tracking studies or semi-repeated annual projects, the task rightly takes a higher priority and requires more consideration.
It’s important to consider problems that you may encounter. Below, we look at things that will work highly efficiently and things that may need more preparation. Although we cannot solve every problem, we have a great deal of experience working with customers upgrading to MRDCL and can often find quick ways to implement the transfer of projects.
Often, the least considered issue when switching to MRDCL is the benefit of making the most of the potential productivity gains that MRDCL offers.
MRDCL often offers a choice of ways to handle most tasks. This choice is different from many other products which have one way of performing a task. New users of MRDCL when under pressure to complete projects may find a way to generate the results they need, but not in the most efficient way. We encounter this phenomenon most with ex-Quantum users who often try to ‘translate’ Quantum methodologies to MRDCL. Often, there are better approaches in MRDCL. We offer a free critiquing service to help new users understand the most efficient methods so that you get the most from your MRDCL licence.
A second issue is that when new users have become proficient in MRDCL, they can often work independently although they are part of a team. MRDCL has some tools that are best exposed and give the most benefit by working as a team. Templates are an excellent example of this. Further, custom templates can mean that your senior staff can share projects with less skilled or less experienced staff.
Complementary tools, such as the free Resolve software, also mean you have more choice about which team members work on different parts of a project.
These approaches can contribute significantly to an organisation’s profitability. Again, we can recommend programmes to help with this and have videos that explain the process. By switching to MRDCL, you can usually benefit by changing the way you work even if it is only a slight change.
A decision to take an MRDCL licence will usually entail a cost-benefit analysis (Learn more about our pricing compared to other software). MRDCL users broadly split into two groups. These are those who use MRDCL out of necessity and those who choose to use MRDCL. In this section, we consider these two types of users, the direction MRDCL is going and the price of using MRDCL.
By referring to using MRDCL as a ‘necessity’, we are referring to customers who have complex needs or a work of a particular type more suited to a scripting language. This might include big tracking studies, the need for complex variables, complex weighting and much more. In such cases, the cost-benefit analysis is relatively simple.
The evaluation is likely to focus on comparing the capabilities of the small number of scripting languages available and looking at each product’s features, costs and expected productivity. You may wish to consider hybrid products, but hybrid products are better suited to those with mostly straightforward analysis requirements that occasionally need to stretch slightly further.
Evaluating which is the better product to meet your needs will still need some consideration, of course. In such situations, we consider MRDCL to be well placed. MRDCL has several significant advantages. MRDCL differentiates itself from the other scripting languages by:
Other scripting languages fail to match MRDCL on all or almost all of these criteria.
For general tabulation production work, a cost-benefit analysis is a more complicated task as many products are available, some of which are much less expensive.
In these cases, there is a need for some other motivations. The motivation may be the comfort of knowing that MRDCL will meet any needs that arise. This may be particularly important when operating a data processing service or being in a position where customers dictate the analysis provided.
However, there is usually a need for some evidence that productivity will improve, costs will be lower or that internal processes run more smoothly.
In these cases, consideration needs to cover:
These benefits are harder to evaluate, but we are always willing to discuss them with potential clients as objectively as possible. We do not believe MRDCL is the right solution for everyone.
It may be helpful to explain, at this stage, that we see significant advantages in the right circumstances to using scripting for data analysis tasks.
MRDCL is known best as a cross-tabulation engine. However, we are moving quickly towards providing a scripted research processes engine, which offers even more significant advantages.
For years, MRDCL has had the power to process research data and produce cross-tabulations, but we have broadened our goals and scope substantially….with more to come in 2022.
Automating the process of providing research insights is at the heart of the MRDCL ethos. Our mission is:
We describe the pricing of MRDCL as a competitively-priced premium product. There are less expensive products available, but MRDCL has only reached its productivity and value by constant, long-term investment in development.
Further, MRDCL has a first-class support team and more resources freely available than any competitor, including videos, blog articles, tutorials and webinars.
Our licences are usually sold on an annual licensing basis, although there are discounts for longer periods.
You can purchase MRDCL through ourselves or one of our resellers. MRDC Software is our main reseller in Europe and Asia Pacific.
As the primary purpose of MRDCL is to handle market research data efficiently, this section explains:
MRDCL handles five types of data. These are:
Additionally, MRDCL can import survey metadata (the raw data, the variable and its associated texts) directly from Triple-S and SPSS. Triple-S data is stored in ASCII format; SPSS uses a proprietary format.
MRDCL allows users to read data from any number of data files. This feature has two advantages:
MRDCL is highly efficient at managing tracking studies. Tracking studies in many survey analysis programs become problematic as questionnaires and data maps change from wave to wave. MRDCL has facilities to:
In some cases where there is a big data set, it is impossible to output all the data into one file – this is often true of some of the low-cost data collection platforms. The solution is to output the data from the data collection platform as two or more files. MRDCL has tools to marry each respondent’s data from the files, but it is sometimes more convenient to read data from the separate files. MRDCL allows users to read data from up to four streams simultaneously (in parallel).
MRDCL also has tools to read more complex data sets from multiple streams. An example illustrates this well. For example, you may have a data file containing data relating to 500,000 customers and data file with survey data relating to 5000 of those customers. You may wish to read some customer data from the larger file for the 5000 respondents in your survey.
For years, MRDCL has excelled at handling hierarchical or multi-level data. Some surveys contain hierarchical data. Hierarchical data is present where the relationship between parts of the data are not on a respondent basis. A typical example is a survey carried out among doctors. The doctor gives opinions or information relating to himself or herself, but may also provide information for, say, five patients. This is data relationship is known as a data hierarchy. Similarly, a respondent may answer questions about any travel occasions in the last week. Each trip will relate to the respondent, but there may be any number of trips. There are no limits to the number of levels of data that can be processed using MRDCL. There are three types of hierarchical data, although the principles are the same with appropriate tools in MRDCL.
These are surveys where the data has a hierarchical structure. An example of this might be as follow. You might survey 100 doctors. For each doctor, you might collect data relating to up to 10 patients. For each patient, you might record the dosages and frequency of dose for each drug prescribed. In other words, there are three levels of data – doctors, patients and drugs. In this example, there is a linear hierarchy. Other hierarchies may have different structure and different relationships. MRDCL can handle even the most complex of hierarchical relationships.
Some questionnaires have repeated blocks of questions (often called ‘loops’ in data collection software). An example of this would be eating out occasions in a questionnaire where data relating to each ‘eating out occasion’ is recorded. Similarly, a survey might record the television programmes viewed for each hour time band in a diary during a day or over several days. A further example might be where there is a series of identical questions for, say, five products. Although it is possible to treat this data as standard respondent data, MRDCL reduces the amount of work and processes such surveys as a form of hierarchical data – the levels being respondents and occasions or respondents and products discussed.
A third example of hierarchical data is where there are rotational sections to a questionnaire. A product test is a good example where respondents might, for example, test three products in a random order. Some software solutions resolve this problem by recoding data, but MRDCL has easy-to-use tools that allow you to treat the data as hierarchical data – respondents and products tested.
It is becoming increasingly important to be able to deliver how clients need it. Older software often struggles in this respect as well as products like SPSS, which aim to make their proprietary data format a universal standard. MRDCL has a good range of tools to link variables and data to other systems.
TSAPI is a new initiative announced in 2020 that we wholeheartedly support. TSAPI is an API aiming to connect survey data between other survey platforms and other business systems. Arguably, it is a modern form of Triple-S. When TSAPI is complete, MRDCL will link to TSAPI as well as read dynamically from it.
There is often a need to weight survey so that it is representative of known populations or targets. MRDCL has a full set of tools to apply weighting of all types. You can apply weighting factors that are already present in the data file or, more commonly, use MRDCL to calculate the factors that need to apply. MRDCL handles both target weighting using interlocking cells and rim weighting. MRDCL can also apply pre-weights and combinations of weighting techniques. When applying weighting, MRDCL allows you to check the effective sample size. MRDCL, unlike some software products, also applies weighting correctly to statistical data.
MRDCL additionally supports quantity weighting where you want to scale analysis by a value within the data.
There is a need to apply weighting with care. MRDCL has several benefits over many other competitor products when weighting is in use:
We regularly claim that MRDCL is more powerful than any other scripting language. We argue that MRDCL offers more flexibility, more automation and more productivity than other scripting languages, but let’s look at this in more detail. There are two types of advantages to MRDCL.
Firstly, there are some features that you could bracket under ‘completeness’. In other words, MRDCL does everything you might want, has no limitations and does it efficiently. We would consider these the ‘obvious advantages’ of MRDCL. It doesn’t mean that other products cannot match some of these benefits or come close; it means that MRDCL has ALL these benefits.
Secondly, some features are unique to MRDCL, which if you utilise well, can make your whole data processing operation faster, more cost-effective and more capable than your competitors. These features can be left unused, but, if implemented well, can revolutionise your capabilities. These features need management and user buy-in but come with big rewards. We discuss this separately below under the heading ‘Optimising your data processing’ below.
MRDCL rightly claims to offer anything you want to do when producing cross-tabulations from market research surveys. This claim covers many aspects of creating cross-tabulations. This includes (but is not limited to):
The second type of benefits is not suited to a list of bullet points. A more business-minded approach is needed to harness the real power of the tools that can optimise your data processing. The first step is to appraise your current methods and explore what things would improve productivity. This is likely to be a different list for each company, where improving productivity would have a real impact. MRDCL achieves this by allowing users to build custom interfaces and custom templates to maximise efficiencies.
However, it extends beyond improving productivity. While templates can reduce the amount of time that staff spend on project specifications, there are some less obvious, even hidden, benefits. It can mean that you develop processes which other teams can utilise to improve productivity and throughput, while reducing errors.
These are significant advantages, but let’s look at two reasonably simple case studies to explain how you can optimise your data processing.
The problem: A client came to us and explained that their coding team provided code lists to the data analysis team in Microsoft Word. The data processing team would copy and paste these texts into MRDCL, adding any relevant syntax. The data processing team produced tables for each open-ended question and passed the tables to a researcher. The researcher would inspect the tables and mark up on the tables which codes they wanted to put together for sub-totals (or nets). The researchers wanted the tables ranked on sub-totals with codes making up each sub-total ranked within it – something which most scripting languages can handle but is cumbersome to specify. The data processing team would then put in the MRDCL code to specify the nets and send the tables to the researcher. Sometimes, the researcher might make some alterations having seen the tables. The data processing team would repeat the process and re-send tables to the researcher. It was a long-winded process for something that looks a simple task.
The thinking: Our immediate thoughts were that the data processing team were spending too much carrying out what other teams could do more efficiently. Indeed, the coders were responsible for the code lists and the researchers for any sub-totals/nets. All the data processing team wanted was a document with all the information in one place.
The solution: Rather than using Microsoft Word, the coders used an Excel template that codes and texts in two columns of each spreadsheet. Each spreadsheet stored the code lists for a specific question or questions. Using MRDCL, the data processing team merely set a reference to the workbook and the worksheet for each question. When the researchers wanted to specify sub-totals, they used a notation in the third and fourth columns to indicate which codes needed to form part of a sub-total. As the data processing team had a custom template that read this spreadsheet in MRDCL, the data processing team had to do no more work to get the tables ranked on sub-totals and the codes within the sub-total. It was automatic. If the coders added codes or the researcher changed their mind about the sub-totals, they amended the spreadsheet and the tables automatically updated. The client claimed that it saved hundreds of hours each year!
The problem: A client identified during a workshop that they produced 16 different types of tables from rating scales. Many of these types were only subtly different, but they recognised that some table types and complex summary tables took time to specify and, sometimes, get right. For example, they wanted:
The thinking: A template where someone could tick off the options that were wanted for a particular set of rating scales was needed.
The solution: There was some input for the data processing team in this case, but it was mainly a matter of ticking which of the 16 options were needed for a particular set of rating scales. There was a need for additional settings and options, such as the variable names, the question and response texts. However, putting it in a template meant that a junior member of the team could specify all the required analyses.
The challenge is to look objectively at the work you have to carry out in MRDCL and explore ways to put it into a template to reuse it from project to project and among all team members.
Focusing on the things that take the most time, are most commonly needed or most prone to error is the key. Simplifying tasks so that less skilled staff (or the right person) is handling the relevant work will significantly improve efficiencies.
Software solutions that work in isolation are becoming less and less valuable. For a product to work in isolation, the product must cover everything you will ever want to do with the data you are handling or processing. The truth of the matter is that this is becoming less likely as the paths of data expand. Consequently, we are opposed to products that do not make it easy to transfer data between applications. MRDCL has many ways of connecting to and from other systems
MRDCL reads survey data from most products. The imports cover most of the products used in market research. Further, you can feed in external business data to analyse it alongside survey data. The best imports for survey data into MRDCL are from the Triple-S standard or SPSS format. We have added features to control the import data from these sources to minimise the work you have to undertake within MRDCL.
Many products, even some of the low-cost data collection tools, will output data in Triple-S or SPSS format. These two formats mean that both data and texts are readily available in MRDCL as efficient MRDCL script and immediately accessible data. Alternatives are as Excel, CSV format or as ASCII. MRDCL can process all these types, but you will need to input or import texts by some other method. If texts are available in a template, it should be possible to use an MRDCL template to read them in.
It is possible to build MRDCL into a system to link data or use MRDCL as an engine. (Read our blog article on integrating MRDCL with other systems). We have implemented some systems that can reduce the time our clients spend getting data and texts into MRDCL. Building simple or complex systems to automate this process is often practical, depending on the source software’s flexibility.
MRDCL has all the tools you need to connect data to other systems. This connectivity may be to supply data to a client database, to prepare data for a dashboard or other purposes. Besides the standard outputs and exports available from MRDCL, it is generally easy to build a custom link to some other system.