The Truth, The Whole Truth and Nothing But the Truth

“The truth is rarely pure and never simple,” Oscar Wilde famously pointed out.

In the audience measurement business a less famous British commentator, Rodney Harris, noted in the 1980s that:

“Media research is not designed to find out the truth. It is a treaty between interested parties.”

Certainly, if by truth we mean perfection, then audience measurement falls down. There are no perfect audience numbers; all are estimates. Short of interviewing everybody in the population and achieving error-free recall or tracking of behaviour, all measurement systems are imperfect.

So we employ statistical techniques to help us. Four stand out: sample weighting, ascription, data fusion and modelling.

It is true that we can tell what blood type a person has or whether they are afflicted with certain conditions from examining very small amounts of blood. And it is equally true that we can learn a lot from survey samples about peoples’ opinions and behaviour.

But sampling people is not exactly the same as taking a blood sample. People are not homogenous. Our samples need to be representative across a range of demographic and other characteristics that influence whatever behaviour we are measuring.

But it can never be flawless. Reporting samples may be skewed towards certain groups which are easier to recruit or who respond more readily. To the extent these biases are known, they can be compensated for by various kinds of sample weighting.

Then there is the issue of information ‘gaps’, where some people have not answered every question or reported on all the days they were asked to report on. Here, answers can be ‘ascribed’ or filled in on their behalf using information we have collected both from them and similar respondents. This is common practice in many audience measurement studies.

Where we want to collect more information than it is reasonable to ask a single group of people to provide, we can use statistical ‘fusion’ to join together different surveys.

Similar participants from each study will first be matched on as many criteria as possible (e.g. on gender, age group, region and any other characteristics considered pivotal to the behaviours being measured). Survey data on each matching respondent is then merged so that we can look at the answers to questions asked on both studies in a single database.

Again, this is common practice. Many readership surveys fuse data from separate online measurement services to report on cross-platform audiences for newspapers and magazines. Out of Home measurement integrates travel survey data with traffic flow information. Increasingly, TV audience measurement combines panel data with internet traffic data to report on total video usage from all sources.

Which brings us to the ‘modelling’ of behaviours. Nielsen, for example, has recently decided to ‘assign’ rather than collect viewer demographics in several US cities. Here, TV set usage is captured automatically by ‘set-meters’ (which detect when the set is switched on and which channel is it tuned to) while a separate sample of people used to fill out a one-week diary to tell us who was watching. …

… read on at ipsos-mori.com

Originally posted by Andrew Green on the Light Bites Blog at Ipsos MORI
27th January 2016

Andrew Green will be chairing a session at the 2016 asi APAC Television Conference in Singapore on 12th-13th May