The science of the sleeper

The decline of the independent bookstore and of hand selling has led to domination by the blockbuster. But what if there were a simple way to recreate the personal recommendation? The Americans, using IT, have come up with just such a concept. Malcolm Gladwell reports on collaborative filtering</p><p>
In 1992, a sometime actress named Rebecca Wells published a novel called Little Altars Everywhere with a small, now defunct press in Seattle. Wells was an unknown author, and the press had no money for publicity. She had a friend, however, who spent that Thanksgiving with a friend who was the producer of National Public Radio's "All Things Considered". The producer read the book and passed it on to the show's host, who liked it so much that she put Wells on her programme. That interview, in turn, was heard by a man who was listening to the radio in Blytheville, Arkansas, and whose wife, Mary Gay Shipley, ran the town bookstore. He bought the book and gave it to her. She loved it, and, with that, the strange and improbable rise of Rebecca Wells, bestselling author, began.</p><p>
Blytheville is a sleepy little town about an hour or so up the Mississippi from Memphis. Mary Gay Shipley's bookstore, That Bookstore in Blytheville, on a meandering stretch of Main Street, is just one long room in a slightly shabby storefront, with creaky floors and big overhead fans and subject headings on the shelves marked with Post-it&reg; notes. Shipley's fiction section takes up about as much shelf space as a typical Barnes &amp; Noble devotes to, say, homeopathic medicine. That is because Shipley thinks that a book buyer ought to be able to browse and read the jacket flap of everything that might catch her eye, without being overwhelmed by thousands of choices.</p><p>
Mostly, though, people come to Mary Gay Shipley's store in order to find out what Mary Gay thinks they ought to be reading. And in 1993 Mary Gay Shipley thought people ought to be reading Little Altars Everywhere. She began ordering it by the dozen, which, Shipley says, "for us, is huge". She put it in the little rack out front where she lists her current favourites. She wrote about it in the newsletter she sends to her regular customers. "We could tell it was going to have a lot of word of mouth," she says. "It was the kind of book where you could say, `You'll love it. Take it home.'" The number one author at That Bookstore in Blytheville in 1993 was John Grisham, as was the case in nearly every bookstore in the country. But number two was Rebecca Wells.</p><p>
Little Altars Everywhere was not a bestseller. But there were pockets of devotees around the country, in Blytheville; at the Garden District Book Shop, in New Orleans; at Parkplace books, in Kirkland, Washington, and those pockets created a buzz that eventually reached Diane Reverand, an editor in New York. Reverand published Wells' next book, Divine Secrets of the Ya-Ya Sisterhood, and when it hit the bookshelves, the readers and booksellers of Blytheville, the Garden District and Kirkland were ready.</p><p>
"When The Ya-Ya Sisterhood came out, I met with an instore sales rep from HarperCollins," Shipley says. "I'm not real sure he knew what a hot book this was. When he came to the store, I just turned the page of the catalogue and said, `I want one hundred copies', and his jaw fell to the table, because I usually order four or two or one. And I said, `I want her to come here. And if you go anywhere, tell people this woman sells in Blytheville.'"</p><p>
Wells made the trip to Arkansas and read in the back of Shipley's store; the house was packed, and the women in the front row wore placards saying "Ya-Ya". She toured the country, and the crowds grew steadily bigger. Ya-Ya sold 15,000 copies in hardcover. The paperback sold 30,000 copies in its first two months. Diane Reverand took out a single-column ad next to the contents page of the New Yorker, the first dollar she had spent on advertising for the paperback, and sales doubled to 60,000 in a month. It sold and sold, and by February of 1998, almost two years after the book was published, it reached the bestseller lists. There are now nearly three million copies in print. Rebecca Wells, needless to say, has a warm spot in her heart for people like Mary Gay Shipley. "Mary Gay is a legend," she says. "She just kept putting my books in people's hands."</p><p>
Sleepers and blockbusters</p><p>
In the book business, as in the movie business, there are two kinds of hits: sleepers and blockbusters. John Grisham, Tom Clancy and Danielle Steel write blockbusters. Their books are announced with huge publicity campaigns. Within days of publication, they leap onto the bestseller lists. Sales start high, hundreds of thousands of copies in the first few weeks, and then taper off. People who buy or watch blockbusters have a clear sense of what they are going to get: a Danielle Steel novel is always, well, a Danielle Steel novel.</p><p>
Sleepers, on the other hand, are often unknown quantities. Sales start slowly and gradually build; publicity, at least early on, is often non-existent. Sleepers come to your attention by a slow, serendipitous path: a friend who runs into a friend who sets up the interview that just happens to be heard by a guy married to a bookseller. Sleepers tend to emerge from the world of independent bookstores, because independent bookstores are the kinds of places where readers go to ask the question that launches all sleeper hits: "Can you recommend a book to me?"</p><p>
Shipley was plugging Terry Kay's To Dance with the White Dog long before it became a bestseller. She had Melinda Haynes lined up to do a reading at her store before Oprah >tipped Mother of Pearl as one of her recommended books and it shot onto the bestseller lists. She read David Guterson's Snow Falling on Cedars in manuscript and went crazy for it. "I called the publisher, and they said, `We think it's a regional book.' And I said, `Write it down. "MGS says this is an important book."'" All this makes it sound as if she has a sixth sense for books that will be successful, but that's not quite right. People like Mary Gay Shipley do not merely predict sleeper hits; they create sleeper hits.</p><p>
Most of us, of course, do not have someone like Mary Gay Shipley in our lives, and with the decline of the independent bookstore in recent years the number of Shipleys out there creating sleeper hits has declined as well. The big chain bookstores that have taken over the bookselling business are blockbuster factories, since the sheer number of titles they offer can make browsing an intimidating proposition.</p><p>
As David Gernert, who is John Grisham's agent and editor, explains, "If you walk into a superstore, that's where being a brand makes a difference. There is so much more choice it's overwhelming. You see walls and walls of books. In that kind of environment, the reader is drawn to the known commodity. The brand-name author is now a safe haven."</p><p>
Between 1986 and 1996, the share of book sales represented by the 30 top-selling hardcover books in America nearly doubled. The new dominance of the blockbuster is part of a familiar pattern. The same thing has happened in the movie business, where a handful of heavily promoted films featuring "bankable" stars now commands the lion's share of the annual box office. But what if there were a way around the blockbuster? What if there were a simple way to build your very own Mary Gay Shipley? This is the promise of a new technology called collaborative filtering, one of the most intriguing developments to come out of the Internet age.</p><p>
If you want a recommendation about what product to buy, you might want to consult an expert in the field. That's a function that magazines such as Car and Driver and Sound &amp; Vision perform. Another approach is to poll users or consumers of a particular product or service and tabulate their opinions. It is very helpful to hear what an "expert" audiophile has to say about the newest DVD player, or what the thousands of owners of the new Volkswagen Passat have to say about reliability and manufacturing defects.</p><p>
But when it comes to books or movies, what might be called "taste products", these kinds of recommendations are not nearly as useful. Few moviegoers, for example, rely on the advice of a single movie reviewer. Most of us gather opinions from a variety of sources, from reviewers whom we have agreed with in the past, from friends who have already seen the movie, or from the presence of actors or directors we already like, and do a calculation in our heads. It's an imperfect procedure. To predict correctly whether you will like something, the person making the recommendation really has to know something about you.</p><p>
Knowing the customer</p><p>
That is why Shipley is such a powerful force in touting new books. She has lived in Blytheville all her life and has run the bookstore there for more than 23 years, so her customers know who she is. They trust her recommendations. At the same time she knows who they are, so she knows how to match up the right book with the right person. For example, she really likes David Guterson's last novel, East of the Mountains, but she is not about to recommend it. It is about a doctor who has cancer and plans his own death and, she says, "there are some people dealing with a death in their family for whom this is not the book to read right now".</p><p>
She had similar reservations about Charles Frazier's Cold Mountain. "There were people I know who I didn't think would like it," Shipley says. "And I'd tell them that. It's a journey story. It's not what happens at the end that matters, and there are some people for whom that's just not satisfying. I don't want them to take it home, try to read it, not like it, then not go back to that writer." Shipley knows what her customers will like because she knows who they are.</p><p>
Collaborative filtering is an attempt to approximate this kind of insider knowledge. It works as a kind of doppelg&auml;nger search engine. All of us have had the experience of meeting people and discovering that they appear to have the very same tastes we do, that they really love the same obscure foreign films that we love, or that they are fans of the same little-known novelist whom we are obsessed with.</p><p>
If such a person recommended a book to you, you would take that recommendation seriously, because cultural tastes seem to run in patterns. If you and your doppelg&auml;nger love the same 10 books, chances are you will also like the 11th book he likes. Collaborative filtering is simply a system that sifts through the opinions and preferences of thousands of people and systematically finds your doppelg&auml;nger, and then tells you what your doppelg&auml;nger's 11th favourite book is.</p><p>
Collaborative filtering in practice</p><p>
John Riedl, a University of Minnesota computer scientist who is one of the pioneers of this technology, has set up a Website called MovieLens, which is a very elegant example of collaborative filtering at work. Everyone who logs on is asked to rate a series of movies on a scale of one to five, where five means "must see" and one means "awful". For example, I rated "Rushmore" as a five, which meant that I was put into the group of people who loved "Rushmore". I then rated "Summer of Sam" as a one, which put me into the somewhat smaller and more select group that both loved "Rushmore" and hated "Summer of Sam".</p><p>
Collaborative filtering systems do not work all that well at first because, obviously, in order to find someone's cultural counterparts you need to know a lot more about them than how they felt about two movies. Even after I had given the system seven opinions (including "Election", four; "Notting Hill", two; "The Sting", four; and "Star Wars", one), it was making mistakes. It thought I would love "Titanic" and "Zero Effect", and I disliked them both. But after I had plugged in about 15 opinions, which Riedl says is probably the minimum, I began to notice that the rating that MovieLens predicted I would give a movie and the rating I actually gave it were nearly always, almost eerily, the same. The system had found a small group of people who feel exactly the same way I do about a wide range of popular movies.</p><p>
What makes this collaborative filtering system different from those you may have encountered on or In order to work well, collaborative filtering requires a fairly representative sample of your interests or purchases. But most of us use retailers such as only for a small percentage of our purchases. For example, I buy fiction at the Barnes &amp; Noble round the corner from where I live. I buy most of my non-fiction in secondhand bookstores, and I use for gifts and for occasional work-related books that I need immediately, often for a specific and temporary purpose. That is why, bizarrely, currently recommends that I buy a number of books by the radical theorist Richard Bandler, none of which I have any desire to read. But if I were to buy a much bigger share of my books online, or if I "educated" the filter, as allows every customer to do, and told it what I think of its recommendations, it's easy to see how, over time, it could turn out to be a powerful tool.</p><p>
In their book, Net Worth, John Hagel, an e-commerce consultant with McKinsey &amp; Compan>y, and his co-author, Marc Singer, suggest that we may soon see the rise of what they call "infomediaries", which are essentially brokers who will handle our preference information. Imagine, for example, that I had set up a company that collected and analysed all your credit card transactions. That information could be run through a collaborative filter, and the recommendations could be sold to retailers in exchange for discounts.</p><p>
Steve Larsen, the senior vice-president of marketing for Net Perceptions, a firm specialising in collaborative filtering started by Riedl and the former Microsoft executive Steven Snyder, among others, says that someday there might be a kiosk at your local video store where you could rate a dozen or so movies and have the computer generate recommendations for you.</p><p>
Among marketers, the hope is that such computerised recommendations will increase demand. Right now, for example, 35% of all people who enter a video store leave empty-handed because they cannot figure out what they want; the point of putting kiosks in those stores would be to lower that percentage. "It means that people might read more, or listen to music more, or watch videos more, because of the availability of an accurate, reliable method for them to learn about things that they might like," Snyder says.</p><p>
Marketing 'taste products'</p><p>
The really transformative potential of collaborative filtering, however, has to do with the way taste products, books, plays, movies and the rest, can be marketed. At present marketers play an elaborate game of stereotyping. They create fixed sets of groups, middle-class-suburban, young-urban-professional, inner-city-working-class, rural-religious and so on, and then find out enough about us to fit us into one of those groups. The collaborative filtering process, on the other hand, starts with who we are, then derives our cultural "neighbourhood" from those facts.</p><p>
And these groups are not permanent. They change as we change. I have never seen a film by Luis Bu&ntilde;uel, and I have no plans to. But if I were to see "That Obscure Object of Desire" and love it, and enter my preference on MovieLens, the group of people they defined as "just like me" would immediately and subtly change.</p><p>
A group at Berkeley headed by the computer scientist Ken Goldberg has, for instance, developed a collaborative filtering system for jokes. If you log on to the site, known as Jester, you are given 10 jokes to rate. (Q: Did you hear about the dyslexic devil worshipper? A: He sold his soul to Santa.) These jokes are not meant to be especially funny; they are jokes that reliably differentiate one "sense of humour" from another.</p><p>
On the basis of the humour neighbourhood you fall into, Jester gives you additional jokes that it thinks you will like. Goldberg has found that when he analyses the data from the site, and 36,000 people so far have visited Jester, the resulting neighbourhoods are strikingly amorphous. In other words, you don't find those 36,000 people congregating into seven or eight basic humour groups, off-colour, say, or juvenile, or literary. "What we'd like to see is nice little clusters," Goldberg says. "But, when you look at the results, what you see is something like a cloud with sort of bunches, and nothing that is nicely defined." The better you understand someone's particular taste pattern, the less predictable and orderly his preferences become.</p><p>
Collaborative filtering underscores a lesson that, for the better part of history, humans have been stubbornly resistant to learning: if you want to understand what one person thinks or feels or likes or does it is not enough to draw inferences from the social or demographic category to which he belongs. You cannot tell, with any reasonable degree of certainty, whether someone will like The Girl's Guide to Hunting and Fishing by knowing that the person is a single 28-year-old woman who lives in Manhattan. Riedl has taken demographic data from the people who log on to MovieLens, such as their age and occupation and sex, but he has found that it hardly makes his predictions any more accurate. "What you tell us about what you like is far more predictive of what you will like in the future than anything else we've tried," he says.</p><p>
This is a potentially revolutionary argument. Traditionally, there has been almost no limit to the amount of information marketers have wanted about their customers: academic records, work experience, marital status, age, sex, race, post code, credit records, focus group sessions. Everything has been relevant, because in trying to answer the question of what we want, marketers have taken the long way round and tried to find out first who we are. Collaborative filtering shows that, in predicting consumer preferences, none of this information is all that important. In order to know what someone wants, what you really need to know is what they have wanted.</p><p>
Blowing away the blockbuster?</p><p>
How will this affect the so-called blockbuster complex? When a bookstore's sales are heavily driven by the recommendations of a particular person, a Mary Gay Shipley, sleepers, relatively speaking, do better and blockbusters do worse. If you were going to read only Clancy and Grisham and Steel, after all, why would you need to ask Shipley what to read? This is what David Gernert, Grisham's agent, meant when he said that in a Barnes &amp; Noble superstore a brand like Grisham enjoys a "safe haven". It is a book you read when there is no one like Shipley with the credibility to tell you what else you ought to read.</p><p>
Gernert says that at this point in Grisham's career each of his novels follows the same general sales pattern. A new title rides high on the bestseller lists for the first few months, but after that, "his sales pick up at very specific times, notably, Father's Day and Mother's Day, and then it will sell well again for Christmas". That description makes it clear that Grisham's books are frequently bought as gifts. And that is because gifts are the trickiest of all purchases. They require a guess about what somebody else likes, and in conditions of uncertainty the logical decision is to buy the blockbuster, the known quantity.</p><p>
Collaborative filtering is, in effect, anti-blockbuster. The more information the system has about you, the more narrow and exclusive its recommendations become. It is just like Shipley: it uses its knowledge about you to steer you towards choices you would not normally know about. It gives voice to the expert in every preference neighborhood. In short, it has the ability to reshape the book market. When customised recommendations are available, choices become more heterogeneous. Big bookstores lose their blockbuster bias, because customers now have a way of narrowing down their choices to the point where browsing becomes easy again.</p><p>
In the US, of the top 100 bestselling books of the 1990>s, there are only a handful that can accurately be termed sleepers: Robert James Waller's The Bridges of Madison County, James Redfield's The Celestine Prophecy, John Berendt's Midnight in the Garden of Good and Evil and Charles Frazier's Cold Mountain. Just six authors, John Grisham, Tom Clancy, Stephen King, Michael Crichton, Dean Koontz and Danielle Steel, account for 63 of the books on the list.</p><p>
Narrowing the bestseller gap</p><p>
In a world more dependent on collaborative filtering, Grisham, Clancy, King and Steel would still sell a lot of books. But you would expect to see many more books like Divine Secrets of the Ya-Ya Sisterhood, many more new writers, making their way onto the bestseller list. And the gap between the very best selling books and those in the middle would narrow. Collaborative filtering, Hagel says, "favours the smaller, the more talented, more quality products that may have a hard time getting visibility because they are not particularly good at marketing".</p><p>
There seems, in this era of megastores, something almost impossibly quaint about That Bookstore in Blytheville. The truth is, though, that the kind of personalised recommendation offered by Mary Gay Shipley represents the future of marketing, not its past.</p><p>
The above is an edited extract, reproduced with kind permission from the NEW YORKER, of an article that first appeared in its issue of 4th October 1999. Malcolm Gladwell is a staff writer and has his own Website at His book THE TIPPING POINT is published by Little, Brown (US) in March.</p><p>