Can publishing keep its soul in an increasingly data-driven future?

At The Story conference earlier this month week, Jamie Byng, c.e.o. of the independent UK publishing house Canongate, said: "Publishing what the market wants is derivative." It was a comment on a seemingly increasingly data-driven era for book publishers. Departments at major UK and US book publishers are now focused on understanding ‘audience insight’ and access to unprecedented levels of data is helping them to anticipate audience interests. As the American author Seth Godin states: "The future of publishing is about having connections to readers and the knowledge of what those readers want."

So the public gets what the public wants. Simple.

Historically there have always been focus groups in the creative industries in a bid to understand audiences. A focus group shown a pilot of "Dad’s Army" found the programme to be "poor", and the audience report was so negative that it found itself conveniently hidden at the bottom of producer David Croft’s in-tray until the TV series was first broadcast on the BBC. The sitcom went on to run for nine series (80 episodes in total), plus a radio version based on the television scripts, a feature film and a stage show. According to Wiki the series regularly gained audiences of 18 million viewers and is still repeated worldwide.

Broadcasters such as Netflix, however, are both distributors and producers of content, and have access to new levels of audience data without having to rely on the limited reliability of focus groups. Its hit series "House of Cards" was commissioned as a result of understanding the Netflix audience. In David Carr’s New York Times article, ‘Giving Viewers What They Want’ (2013), he reported: "(Netflix) already knew that a healthy share had streamed the work of Mr Fincher, the director of "The Social Network", from beginning to end. And films featuring Mr Spacey had always done well, as had the British version of "House of Cards". With those three circles of interest, Netflix was able to find a Venn diagram intersection that suggested that buying the series would be a very good bet on original programming."

But can book commissioning really be treated like a science in this same way? If it were, then surely Amazon would be at a huge advantage with its unrivalled access to consumer data. According to the New Yorker’s George Packer (2014), Amazon’s initial game-plan of selling books was not just about learning about what books readers liked, but understanding what consumers wanted generally, to learn what else they might sell to them. It would be tough to argue that Bezos’ game plan didn't work on that front.

"Bezos said that Amazon intended to sell books as a way of gathering data on affluent, educated shoppers. The books would be priced close to cost, in order to increase sales volume. After collecting data on millions of customers, Amazon could figure out how to sell everything else dirt cheap on the internet."

However, if you look at Amazon’s foray into book publishing (that is to say, creating and commissioning content - not its Kindle Direct Publishing self-publishing platform) the results are, as Packer puts it, "decidedly mixed".

As Alex Shepherd reported in The New Republic, Amazon's hiring of Larry Kirshbaum in 2011 to lead "Amazon Publishing's East Coast Group" and launch the New Harvest imprint did not go quite as expected: "While Kirshbaum eventually landed a string of reasonably well-known non-fiction authors, including Tim Ferriss and Penny Marshall, the results were mixed at best. Ferriss’s book sold well, but didn't meet expectations and undoubtedly took a substantial hit after Barnes & Noble refused to carry it. Marshall was given an $800,000 advance for a memoir that is now regarded as one of the biggest flops of the 21st century."

According to Shepherd: "Amazon Publishing tried to play the publishing industry on its own turf and lost."

So perhaps books simply can’t be treated like a commodity. Amazon is often cited as being at the root of an attempted commoditisation of books, but academics such as John B Thompson believe this came with a much earlier ‘polarisation of the field’ with the corporations on one side and the indies on the other.  As Helen Bagnall reported for The Literary Platform from Hay 2012, Thompson believes that "the conglomerates also led onto a preoccupation with ‘big books’. Boardrooms prefer growth to literary endeavour.  Book publishing, however, is a mature business. If left to its own devices its growth is fairly static. However, by the mid-90s, book publishing in a corporate structure on both sides of the Atlantic became subject to ‘the growth conundrum’: the need to find an additional 10% revenue year on year to satisfy shareholders. One way to cope with this was for publishers to reduce their lists and concentrate on selling more copies of fewer books, effectively cutting out the mid-list. Another way to plug this 10% gap was to turn to ‘extreme publishing’. This high-concept style publishing crashed books which were riding on the back of a pre-established buzz into the schedule."

In an ironic twist, it is Amazon’s publishing venture AmazonCrossing that is now identifying niches ‘underserved by mainstream publishers’, committing $10 million to publishing more works in translation.

So is growth truly now a bigger pre-occupation to literary endeavour in book publishing boardrooms, and can audience insight even guarantee this growth?

Book publishers are certainly still pursuing their traditional role of recognising, nurturing and developing talent, and many would argue that it is in shoring up the bestsellers that enables publishers to take risks with its more literary imprints, lesser-known authors and develop its mid-list.

In a bid to better understand the landscape for today's literary fiction writers, Arts Council England is currently looking at how the current climate is impacting on literary fiction, recognising that the existing environment presents challenges. Canelo is working on this important piece of research, and the forthcoming report will consider whether "changing technology, a historic shift in the markets for informational and entertainment goods, and rapidly evolving consumer preferences all mean assumptions that literary fiction is in a precarious place".

Let us hope that a focus on the numbers game doesn’t prevent the next Mantel from breaking through. For many, it is book publishing’s seemingly unscientific nature that makes it so intriguing and characterful. Perhaps it’s for this reason that book publishing seems to be more often likened to gambling than a science. In the meantime, we're going to publishing the Cats and Cucumbers Colouring-In Book - it's got to be a winner.

Sophie Rochester is founder of The Literary Platform and chair of The Jerwood Fiction Uncovered Prize.