Last week the Media Research Group held their annual conference in Berlin.  The organisation is 50 years old and they have stoically refused to change the word research in their name to insight. If you noticed your “insight” department thinned out a bit last week; that’s where they were. 

These are people unafraid of slides entitled “methodology”. There was a whole paper on the methodological effect of online research. There was a rude joke about rim weighting.

The conference featured presentations from the insight departments of The Guardian and News UK’s Insight departments. The very different approaches taken by the two newspapers highlighted just how complex measuring what a newspaper does in the digital age has become:

Complexity One: Newspaper content is available on many platforms. There is more to measure and the number is growing. As research and analytics companies are just getting to grips with mobile, newspaper companies are experimenting with wearable devices. The FT has a smartwatch app while the Guardian has a Google Glass app.

Complexity Two: Newspaper content is appearing in forms less obviously qualified to be part of a newspaper’s cross-platform reach. There are many grey areas. The Guardian encourages developers to use Guardian content in their own applications. Newspaper content appears, unbundled, in social media streams. 

What about your columnists tweeting under their own names? Is that part of your reach or not? If a person attends a live Guardian event in a Guardian owned event space should they be part of your reach (because someone watching the footage on the Guardian website next day sure is)?

The Guardian’s G360 research project shows just how challenging this makes it to track everything a newspaper produces. It covers 30 Guardian content areas on 11 different platforms (including 3rd party apps). Working on the project, analysts Kantar Media used a mix of methodologies, incorporating 10,000 face-to-face interviews, an online panel of 1,426. It carried out 4,112 online interviews with Guardian readers and re-contact diaries.

That’s four methodologies just to quantify the way content from one newspaper is read.

Complexity Three: We are going to have to get used to mixed methodologies. A much used quote at the conference was attributed to media research figure, Richard Marks:

“Media researchers are moving from being farmers, lovingly designing and tending their own samples, to being chefs, creating insights from high quality ingredients.”

Mixed methodologies are also harder for the people using that information to understand, especially as they require modelling, fusion and advanced statistical techniques.

Complexity Four: One experienced delegate told me that two decades ago it was common for senior people from media agencies who were not researchers to attend the conference. Perhaps that belongs to a different era when there were fewer media conferences, larger expense accounts and larger teams back home to delegate to. It is a shame because methodologies are becoming more complex but the ‘how and why’ conversations are being held only between researchers.

Complexity Five: Researchers are finding they need to introduce new metrics to explain how content works on new platforms. News UK head of insight Sean Adams discussed the way Times tablet app readers read The Times vs. the way Times print readers read their newspaper. On the surface Adams found many similarities. Both groups recalled the same number of articles and they often remembered the same specific articles too. 

Adams borrowed a metric from neuroscience to help him draw out differences. Memory encoding, he told us, is the most important description of engagement because it describes the way information ends up in our long-term memories. It is driven by personal relevance, emotional intensity and narrative. It is measured via electrical activity in the relevant parts of the brain. 

Reading via a tablet triggers more electrical activity so it encodes into memories faster than print. Print is a less stimulating though less tiring way to read and so it ends up encoding as much into our memory as tablet reading does but over a longer period. What this means, says Adams, is that dwell time isn’t a reliable measure of engagement. Less dwell time on a tablet is as good as longer dwell time with a newspaper. 

That’s a hard message for the industry because dwell time sounds intuitively right as a measure of engagement. In fact the Financial Times’ attention minutes initiative, which makes a virtue of dwell time, has attracted interest and praise. Only chunks of time where an ad is in full view for more than five seconds can count towards the hour-long blocks the FT is offering advertisers in cost-per-hour deals.  

The FT believe that ads in-view for 5 seconds or more deliver double the favourable responses (recall, purchase intent) compared to those in view for less time.  Add Adams’ finding to that thinking and a likely conclusion might be that what is five seconds for one platform might be just three seconds on a different platform that triggers more electrical activity yet commands less dwell time. 

In previous years, others have talked on similar themes. In the mid naughties Robert Heath delivered an MRG paper on behalf of News International in which he described how he’d measured media consumption using the metric ‘fixations’.  Some mediums trigger more ‘eye fixations’ than others.  He quoted the German psychologist Kroeber-Riel, ‘Eye fixations… serve as a behavioural measure of information processing’

Two things have changed since then. It has become less expensive to measure electrical activity and fixations because kit  (eye cameras etc.) is cheaper and methodologies have evolved. Heath had conducted a handful of interviews; Adams based his finding on 150 in the quantitative part of his research. It would, however, be expensive to extend it to ten, fifteen, twenty plus platforms.   It would be more costly still to compare the content of different newspapers across those platforms.

Secondly, the imperative is greater. Tablets and smartphones are plentiful now and electrical activity and eye fixations seem well suited to drawing out their potency. Heath had used his mid 2000s paper to highlight the advantages of old school print over old school television. Quaint. 

Yesterday seems quaint and today seems complex. Mixed methodologies are now unavoidable as new platforms and formats for newspaper content add to the complexity. New metrics might be required but they might be hard to measure at scale and are potentially less intuitive than old favourites like dwell time.

But getting to grips with this complexity is all the more important because it reflects the reality newspapers are dealing with. Those attending the conference in Berlin may be the experts, but everyone working in newspapers needs to understand how what they produce is being consumed. 

Update 23/11/2014: This story has been amended to replace a description of memory encoding being measured in ‘flicks’ with more specific references to electrical activity in the brain.

Neil Sharman is a research and insight consultant who has worked with media brands including News UK, the IAB and Centaur. You can find his website here and blog here.