Forget AR and VR - mixed reality is the next frontier for digital publishing

Forget AR and VR - mixed reality is the next frontier for digital publishing

While attending TECH(K)NOW International Women’s Day conference earlier this year, I couldn’t help but notice a group of people clustered around someone wearing a lensed headset, who was trying to grab invisible objects in the air. But this wasn't VR... it was mixed reality on show.

What?

A mixed reality headset may look much like a VR one, but the two technologies are very different. MR doesn't offer complete relocation into a separate digital space, but mixes the physical with the digital - much like augmented reality, but in a more thorough and immersive way. The cameras and sensors incorporated onto the MR headset scan the environment and create a 3D map of the user’s surroundings so that the device knows where and when to insert digital objects in real space, which can then be manipulated and interacted with by the user.

As I tried the lenses on and was immersed into my very first mixed reality experience, I couldn’t help but think about the potential such technology could have for the book industry. So I sat with Becky Jones, founder of the mixed reality studio Bektio, who was showcasing the new technology, to get her take.

"Imagine you’re watching a Rihanna concert at home, but instead of watching her on your TV or laptop, she is standing in the middle of the room singing with you," she explained. "Perhaps you’d like to transform your table-top into an airport to view flight paths of air traffic, or maybe when studying the solar system you’d like to walk around a scale model and then zoom out to view the entire galaxy. These are examples of what is feasible with mixed reality technology." 

MR has not yet been used for narrative content, but looking at what other industries achieve with MR gives a good insight into the possibilities for digital publishing. It is, for example, a very popular technology in creation and design industries like architecture and construction. Product designers can create 3D models (a car, for example) and view them instantly, life-size, in order to check the design before being sent to production. Architects can use MR to pre-visualize their plans in 3D and at a tabletop scale, or else go on site and see and explore a human–scale prototype of their building. It is also a great tool for collaborative design as the lenses allow multiple users to interact in the same MR space. Training and development is another key area of use for MR - medical educators can use 3D life-size virtual bodies to teach anatomy, practice procedures or investigate MRI scans. 3D catalogues showing replicas of products are also a strong commercial use for MR that could be implemented in various retail industries - like bookshops, perhaps. The next technical step for MR is to add the volumetric video – a technology that will allow to feel virtual objects and navigate with gestures and voice in order to give a total sensory experience.

So what can be done with books? Jones tells me that the technology has potential “anywhere that overlaying information or providing extra content on top of the real world around you may be useful", so reference or contextual material has instant potential for an upgrade. "The 3D visual exploration is very well suited for educational content and could become the next step for the interactive digital book". 

There are also possibilities in terms of narrative. A MR headset could, for example, recognize where you are in a story and animate relevant setting and characters. “Imagine reading The Hobbit, turning the page and having Smaug the dragon fly into the room, land to your feet and remaining there for the rest of the chapter,” Jones says. Examples like this will resonate with the children’s book industry; Carlton Press is already using augmented reality to pop dinosaurs out of your tablet while you read books in their Digital Magic series.

There's also a chance for adventurous authors to open up their stories to readers - whether by choosing to add extra characters or deciding which way the narrative unfolds. It could become a virtual, audio-visual version of Wattpad, the popular platform that allows readers to comment and add to stories, becoming something like co-authors of each book.

The mobile storytelling platform Oolipo, which was a finalist in the 2015 FutureBook BookTech Award, is probably the closest existing application of MR for digital authors available now. The app mixes three visual modes of interaction, each allowing you to approach content in a different way. It has been described by users as a mix between Netflix, Youtube and Snapshat, and the next step for Oolipo is to let users create content in real time.

There are interesting opportunities to transform reading, writing and selling spaces, too.  “When reading a physical or a digital book the world around you could change to fit with the environment you’re reading bringing you in the appropriate context," Jones explains. "You could choose to be sat on a beach dialling up and down reality around you giving you a calming meditative space better suited to your reading style.” A reader could change their living room into a library of their own design and browse books on personalised virtual shelves instead of on a corporate website.

We won't see readers wearing MR lenses for a while; the technology is still very young and a Microsoft Hololens headset costs £3000. However, more and more makers are expanding into MR devices like Meta, Magic Leap, Google Tango and Apple, which is considering adding a depth sensor to the iPhone. An MR-enabled smartphone can't be far off.

In other words, publishers should be exploring the possibilites of MR now. It will take money, time, collaboration and courage. But the leaders in this young field will undoubtedly reap the rewards in years to come.