Using MRDCL with other software products

The importance of being able to move market research data easily between different systems has never been greater. If a software supplier tells you that their platform does everything you will ever need, be very suspicious. To be fair to (some of) these software suppliers, it might do everything you need, but that overlooks the fact that your client might want the data in some other form or another platform. And, this demand will only increase. MRDCL strives to make data available in other software easy.

Broadening the scope of market research

It doesn’t stop there. Yes, it’s important that software systems for market research can “talk” to other software and systems, but it goes further. Market research software needs to work with external data, such as sales data, targets, KPIs, and other business data. I would argue that the future of market research depends on this progression. It’s something that Kristin Luck, the new ESOMAR President, highlighted in a recent NewMR webinar. Importantly, it offers an opportunity to get market research data in the boardroom.

Compatibility with other software a necessity

My position is that being compatible with other systems is a necessity. However, I go further than that. You may note in the first sentence of this article, the word ‘easily’ appeared. That’s crucial. It has to be easy. By easy, I mean that it takes less than half an hour and is preferably a one-button click. I saw a software presentation recently where the presenter told us that you could deliver data to any system in the world. Passing over my doubt about the presenter’s claim of ‘any system’, the word ‘could’ was the one that jumped out at me. The presentation demonstrated that you could use a programming language to generate data exports. A complex tracking study might need a week or more’s programming to output data from what I could see. That’s not good enough.

How does MRDCL compare?

If you think this article is just a veiled attempt to promote MRDCL as MRDCL is the ‘master of data connectivity’, you would be wrong. MRDCL is good when it comes to data connectivity, but it’s not perfect. What this article will tell you is that we take the ability to transfer data to and from software products easily (remember that word ‘easily’) as highly important. The rest of this article, therefore, will look at different types of data connectivity. For each item, I will comment on where MRDCL stands right now. Let’s start with the most common ways of transferring data and then move on to the types of data transfer you might need.

MOST COMMON FORMATS FOR TRANSFERRING DATA

1. Triple-S

In the early 1990s, the Triple-S data interchange format was born. It started as text-based but moved to XML in 2000. Arguably, it was ahead of its time and mainly allowed users to transfer project data from one market research software package to another market research software package. As a result, a high proportion of software suppliers adopted the standard. Although there have been updates to the Triple-S standard since 2000, there has been no activity since 2017.

Pros

• It is not a proprietary format; the ASC, an independent organisation, endorses the format
• The interchange format focuses on market research data
• The structure contains both the respondent data and most of the text associated with a survey

Cons

• It is only suitable for market research data and not applicable to other business or commercial data
• Few software vendors outside mainstream market research have adopted the standard

MRDCL’s rating

• Imports from Triple-S to MRDCL are of a high quality and can be rerun easily
• Exports to Triple-S to MRDCL are of a high quality and can be automated

2. SPSS

SPSS files are a proprietary format that exposes an object model that other software products can access. Importing data from SPSS can be problematic as it natively stores multiple response questions as one field per response. As a result, imports from SPSS often require further work for most products, although this is one area that MRDCL has recently conquered.

Pros

• It is becoming the most common export for the ever-increasing number of online data collection platforms
• Unlike most software products, it is used widely by both market research agencies and other businesses
• It has an object model that makes it possible to develop automation systems

Cons

• It is a proprietary format
• It does not handle multiple response questions well

MRDCL’s rating

• The new MRDCL module, MRDCL Central, has a high-quality import from SPSS, allowing you to produce variables and an MRDCL script that is efficient and ready-to-use
• MRDCL allows you to export files to SPSS as SAV automatically

3. TSAPI

Strictly, TSAPI should not be under the heading ‘Most common ways to transfer data’ as it is too new. However, in my view, it warrants comment. TSAPI, according to its website, describes itself as ‘The New API standard for transferring survey data’. Yet, silence has followed its launch in mid-2020. Potentially, this is a game-changer for market research and offers the opportunity to merge market research data with other business data. However, it’s not clear what the path forward is or who is leading it. Thus, there is a need for a big marketing effort to get TSAPI accepted in the wider world and persuade market research software vendors to link to its standard.

Pros

• Unprecedented opportunity to bring survey data and business data closer together, enabling the market research industry to deliver a better product
• APIs are the modern way to connect data between software products

Cons

• The development is in its infancy
• There needs to be some clarity as to who/what is promoting TSAPI
• A big marketing effort is needed
• Further development is needed as well as documentation

MRDCL’s rating

Like almost every other software vendor, MRDCL is waiting to see where this initiative goes. Potentially, it is very exciting for the whole market research industry. However, adopting the standard within and outside the market research industry is necessary to succeed. Therefore, MRDCL is ready to move as soon as its road map is positive.

4. Comma-separated files (using codes or texts)

Comma-separated values (CSV) files are a common way to move data from one system to another. Usually, the first line or row of the file contains the identifiers for each field and subsequent rows store the raw data for each record. The raw data may appear as codes or texts. The disadvantage of this method is that there is usually additional work to do after importing. This includes labelling variables and pulling together multiple-response questions.

Pros

• Easy to understand
• Viewable in Excel / Google Sheets

Cons

• A lot of manual labelling of variables will usually be necessary
• You may have difficulties with multiple-response questions
• Other additional work to structure imports conveniently is likely

MRDCL’s rating

In the 2021 release of MRDCL, we allowed users to read CSV data directly rather than using a clumsy workaround. However, users still have to attach labels to responses, so variables are not ready for analysis without additional work.

SPECIFIC TYPES OF DATA TRANSFER

Online data collection platforms

There is a huge range of offerings when it comes to online data collection platforms. They vary in price from free/low cost right up to tens (in some cases hundreds) of thousands of dollars to use each year. Some only collect data; others are complete systems, which include tabulations and data visualisation. Some only use a graphical user interface (GUI); others use a programming or scripting language; some use both a GUI and a language. Many end-to-end systems only cover everything to a basic level; others allow you to program whatever you want within reason.

Every data collection platform is different

For these reasons, making generalised comments is difficult. The design of some platforms is for everything from data collection to online reporting/data visualisation to take place in one system. This type of software tends to sound alarm bells to me. It is fine to use an end-to-end system to handle simple projects or projects that conform to a style. For research agencies, the likelihood is that more advanced specialist software will be necessary on occasions for part of the end-to-end process. For that reason, even the all-in-one systems should make it easy to transfer data to wherever you need it.

Data visualisation platforms

Data visualisations are mostly less problematic for an individual project. This is because they are typically at the end of the process, so there is less need to connect anywhere else after importing data. Many data visualisation platforms are good at presenting data but lack the depth of analysis that you may want. For this reason, data visualisation platforms should enable users to transfer data easily.

External databases

Delivering data in a convenient form to buyers of research is becoming increasingly important. Unfortunately, there are not many platforms that make this easy and convenient to do. I think this will become a requirement over the next five years.

Good/bad connections: A story

Finally, I would like to discuss the topic of good and bad data connections. I will end with a short story. I witnessed a customer who received data for a tracking study weekly from the fieldwork company. The data processing department set up a system to read the data from SPSS and run tables in the tabulation software. No one worried about the actual process until one week the data arrived late. The data processor worked hard to make up for the lost time by working longer hours than usual. The fieldwork company then spotted an error in their data and re-supplied the data a day later. Once again, the data processor worked extra hard but, under pressure, made a silly mistake resulting in delivering wrong results to the client.

Why did things go wrong?

The business owner spoke sharply to the data processor and threatened him with dismissal for making such a bad error on an important client’s work. What was my opinion of this incident? Well, the fieldwork company were late, but the software used was the problem. The connection between the data collection platform and the data analysis software was poor. The fieldwork agency’s software was too inflexible, and the analysis software was too limited to control the import effectively. Better software could have reduced the time taken to less than one hour – and that includes checking.

Was that story a sales pitch for MRDCL?

Now, you may be thinking that I chose this story as a lead-in to a testimonial. But, as it happens, the story did not involve our software at all. Instead, it comes from ten years ago. An independent software developer I knew at the time developed a way of converting files from one format to another effectively. It was what developers call a ‘hack’ solution, but it worked well.

Conclusion: The effect on MRDCL

The way it did affect MRDCL was to make us realise that data transferability was important. Not only that, speed, ease of execution and a low risk of errors were crucial. It has been one of the guiding principles of developing MRDCL, and we will continue to pursue excellence in this area. Data transferability is not just desirable; it’s a necessity.