Neal Stephenson & Co. turn failed Magic Leap AR project into an Audible drama

Neal Stephenson, the science fiction novelist, has been a idol of mine for a while. I loved reading his Snow Crash, the sci-fi book that introduced the concept of the metaverse (the universe of virtual worlds that are all interconnected) back in the 1990s. And I happen to be listening to his dense sci-fi novel, Anathem, right now on Audible. So you can imagine my surprise when former Magic Leap chief creative officer Graeme Devine introduced me to Sean Stewart, an alternate reality games expert; Austin Grossman, game developer and novelist; and — yep — Stephenson about an Audible project they were working on.

Their nine-hour audio drama is being published today. It’s called New Found Land: The Long Haul, and it’s something that they conceived while working on a big project on augmented reality at Magic Leap. They had always intended this audio drama, which is like a radio play with actors, to be a standalone Audible experience. But they conceived of the backstory while working on an augmented reality application for the Magic Leap One AR headset. Sadly, the fortunes of Magic Leap did not allow them to finish their AR project, which would have overlaid imagery from two alternate universes on the real world.

I was delighted to interview Stephenson and Stewart about the audio drama and how they worked on it until their division was laid off, when Magic Leap pivoted away from consumer technology and focused on the enterprise. They are all veteran storytellers who came to the project from different directions — sci-fi novels, video games, and alternate reality games.

But I’m not entirely sure how much of their story I can believe. It is fiction interwoven with reality, as there is a Magic Leap character who is part of the audio drama story, which is like a radio play with actors. But both Stephenson and Stewart were adamant that I understand that there is no alternate reality game to go with this audio drama. They don’t want people to start hunting down clues on the web or the real world and find out that supposed bread crumbs actually lead to nothing. Stewart knows how obsessive people on the internet can become, as he worked on the ilovebees.com alternate reality game when he was at 42 Entertainment. That was a real-world and web-based set of puzzles that led to a marketing campaign for the launch of Halo 2.

They insist it is a stand-alone work of fiction, but they also did not verify to me whether they truly believe there are two alternate realities to the one we live in. For now, I’m going to take them for their words when they say they aren’t putting me on. And you should, too, so don’t waste a lot of time thinking there’s an alternate reality game to go with this audio drama. I’m going to listen to this audio drama, after I finish about 10 more hours of listening to the 32-hour Anathem novel that I’ve started. Heck, I think someone ought to go ahead and make the AR app for these guys.

Here’s an edited transcript of our interview.

Above: New Found Land: The Long Haul is an audio drama on Audible.

Image Credit: Audible

GamesBeat: The piece has a very cryptic description. Where did you want to start as far as trying to describe it?

Neal Stephenson: What have you heard so far?

GamesBeat: Just that it got started at Magic Leap, and then it turned into this 9-hour audio drama. It sounds like there’s a web introduction to it out there as well that you’ll have going soon. That’s about it.

Stephenson: This emerged from a group that I was running out of Magic Leap’s Seattle office between around 2015 and 2020. It didn’t start this way, but it ended up containing three novelists and a bunch of other people from various branches of the game industry. The idea was to create an IP, a story universe, that was well-adapted to augmented reality.

There are two ways we can play this. We have a fictional premise that we’ve been going with, which is that we’re a couple of optical scientists at Magic Leap who got assigned the job of tracking down some audiovisual glitches appearing in an earlier version of the equipment. The more they investigated these glitches, the more they saw that they weren’t random static, but actually glimpses into a different world. They created a device called the PHILTR, which is the Phase Hopping InterLaminary Transmission Rejector, which is a way of eliminating the crosstalk from these other universes, but it had a switch on it. It could switch from the rejector setting to the resonator setting.

When it operates as the resonator, it’s like a crystal radio that picks up transmissions from these other universes. If you hook it up and plug your Magic Leap device into this thing, which they build into an old tin lunch box, you can see these other worlds. There are two of them. One is called Laminar, and the other is called Old Gnarly. They split off from our timeline — or our timeline split off from them, depending on how you look at it — we split off from Old Gnarly about 1,000 years ago, and it became a version of our world dominated by magic. We split off from Laminar about 100 years ago, and it became an alternate timeline that’s dominated by the kinds of big cool analog technology that we never got. They have the flying cars, the jetpacks, the blasters, but they don’t have digital technology or networks. They have Mars colonies, but they still use paper telephone directories.

We created these two fictional Magic Leap employees and wrote them into the story. The first drop was from 2018. It was a series of YouTube videos in which an agent gets sent to our world from Laminar to investigate this man and this woman in the Seattle office of Magic Leap who are building this PHILTR device. Hijinks ensue. That all culminated in an ARG, the kind of thing that Sean is a world-leading expert in, at the New York Comic-Con in October 2018. While we were there, we had breakfast with a producer from Audible. That’s when we had the idea of creating an audio drama that would tell the next phase of the story. That’s been in the works ever since.

We were all let go from Magic Leap in April of 2020. The entire creative studio part of Magic Leap was disbanded. But that didn’t prevent us from being done, because Audible was going to produce the whole thing anyway. They already had the only thing they needed from us, which was the script, mostly written by Sean and Austin with some contributions from me. They just went ahead and produced the thing. It’s coming out on Thursday.

Sean Stewart: I think it’s safe to say there’s a lot of Neal Stephenson DNA through the world and the script.

GamesBeat: There are pages already up, but it’s not going live until Thursday. Is there another component too? Will Amazon have something, or did I misread that?

Stephenson: Amazon and Audible are joined at the hip. This only exists as an audiobook, though. You can find it through Amazon, but the only way to get it is to buy the audio drama from Audible. You can pre-order now, but you can’t actually get it until Thursday.

GamesBeat: I listened to Anathem, and also to Snow Crash again on Audible. I liked all the music in Snow Crash. Is that an example of what we’re going to see with some of this as well?

Stewart: This is a different deal. This is a full cast. We have 15 cast members. It’s not just being read. It’s literally four Marvel movies with your eyes closed. Sound effects, music, all the characters.

Stephenson: It’s what we used to call a radio play, but you’re not supposed to call it that now. Now it’s an audio drama. This comes from a thing that Audible’s top brass have been wanting to do for a long time, which is to create entertainment that’s written to the form, meaning not just an actor reading a book out loud, but a fully scripted and produced audio drama.

Above: Anathem got started at Magic Leap.

Image Credit: Dean Takahashi

GamesBeat: Did the rights revert to you when the studio disbanded?

Stephenson: They certainly own the rights, yeah. They’re not about consumer-facing entertainment right now. They’ve pivoted strongly toward commercial, industrial types of applications.

GamesBeat: This sounds like a big project, but are you still interested in AR and seeing that come to fruition at some point?

Stephenson: It would be cool. It was made for AR. The IP is calibrated to work well as an AR project, and so it’s definitely doable. We had made some advances toward that here in Seattle. We’d built the beginnings of a location-based experience where you could look out the window of an office suite into Pioneer Square and you could see what Pioneer Square looked like in either the world of Laminar or the world of Old Gnarly and interact with it to some degree. But it’s hard. Making AR content is challenging still. Once you’ve worked on that for a while, making an audio drama or any other kind of standard media seems incredibly easy by comparison.

Stewart: If we could get the IP back or get Magic Leap interested in making the thing themselves, that’s where it all started from. It would be awesome to do. Once you’ve sat in a building in Pioneer Square and watched a zeppelin land and guys with jetpacks come out of it, you want to see that again.

GamesBeat: Sean, I used to be very familiar with the 42 Entertainment people. What did you bring in from the ARG world to this?

Stewart: There are two things. One is, as Neal said, we had an ARG of modest proportions, and the assets for that are still out there waiting to be discovered by people who have the audiobook. Most people who encounter the audiobook will have no idea that there’s an entire prequel out there waiting to be found in the world. And then second, if you know — I was one of the founders of 42. If you know that work, you know that I was the lead writer on I Love Bees, which was a mere 5-and-a-half hour audio drama. We’d been around the block, or at least I had been, writing for this format, which was extremely helpful.

GamesBeat: Is there any of I Love ees in this, would you say? There are things to find in the real world?

Stephenson: To set expectations, there’s not an active ARG running at this time.

Stewart: Right. But in 2018 there was basically a story that unfurled around three pillars. There was a website, a community forum for a group called the Yarnies, who were fans of a golden age comic book that had been erased from the world. If you know the term the Mandela Effect, where you’re totally certain something had happened, except it never happened, but lots of people agree that it also happened — there’s a famous example of a movie with Shaquille O’Neal from the ’90s, except there’s no such film. Anyway, all these people earnestly know that they’ve seen these comic books, but no one can prove that they exist.

There’s a second pillar, which is the optical scientist who is upstairs from us at Magic Leap discovering other worlds. When he looks through the device, he recognizes what he sees, because his flaky would-be comic artist brother used to draw these two worlds all the time. You would take those two pillars of the story, and then the third was this set of YouTube videos by this just-out-of-high-school kid who discovers the agent from another world. Slowly those three storylines track together. There were things that people could do in real life. They got pieces of mail and all the stuff you know about.

Above: I Love Bees for Halo 2’s launch was an ARG.

Image Credit: Microsoft

GamesBeat: Is that sign, where you guys took the picture, a real place? Or is that a set?

Stewart: That is really Enchanted Rock, Texas. We really did go there to do research. There’s a conceit behind the world, which is that basically every couple of hundred years, a meteorite or comet shower passes, dropping — I believe the technical term Neal uses is “magic space dust.” Dropping magic space dust on our planet, which is used to power the spells in the fantasy and the technology in the sci-fi world, but for us, we’ve just never known it was good for anything. It’s like the Comanche riding horses over giant amounts of oil in Oklahoma. We never made the connection.

Enchanted Rock itself is a giant meteor strike from an event related to this comet passing. Neal, because Neal is Neal, has a program in Mathematica which lists every piece of cometary magic space dust that has landed on earth since 10,000 BCE. You can chart it, because of course he does. That’s the pleasure of working with Neal.

GamesBeat: To be clear again, there’s an audio drama that’s finished. There are pieces of an ARG out there in the world, but not finished? And an unfinished AR application.

Stewart: There is no AR application, and we’re not mounting a current ARG with the bells and whistles.

Stephenson: We’re being nervous, because when enthusiasts for the ARG form get going on one of these, they’re relentless. They’ll blow through every problem you can set in a matter of hours. We don’t want them to wrongly get the idea that there’s one of these currently running and then spin their wheels looking for nonexistent clues.

GamesBeat: But this drama, in the fiction, does have Magic Leap employees in it?

Stewart: Magic Leap is never named. They work at an AR startup. That was always the deal from the beginning, that AR would be part of what we were building for, but we didn’t want anyone to come to the work and feel like it was a very peculiar form of product placement.

Above: In I Love Bees, 42 Entertainment made 50,000 pay phones ring at once. Players recorded the calls and put together an hours-long broadcast on the Covenant invasion.

Image Credit: 42 Entertainment

GamesBeat: While you’re being careful here about not creating expectations of an ARG, your fiction is pushing those people in that direction.

Stewart: One hundred percent. But fairly tongue in cheek. One of the values of this kind of conversation — if people go out and start looking for extra stuff, we will gently tell them to stand down. We will communicate that there is archival material to find, but there is no new stuff to do.

GamesBeat: It sounds like it must have been hard to extract something from this, when you expected to have all this technology around it to complete it. Was it difficult to finish the story without that technology?

Stephenson: In the particular case of The Long Haul, the audio drama, that was always planned as what it is from the beginning. It doesn’t require any AR stuff at all.

Stewart: It’s its own story, but the original plan would be that it would also — if you think of the first chapter being told as an ARG and the second chapter being told as an audiobook, then the location-based things that Neal was talking about, that would be the third evolution.

GamesBeat: Was it just the three of you that wound up doing this, plus the cast at Audible?

Stewart: In terms of the audio drama, yes, it was the three of us who were writing it, and then Audible produced it with the actors. Neal had a team in Seattle that included people besides the three of us. They were building, for instance, the location-based experience. For that you need engineers and artists.

GamesBeat: It must be satisfying for this part, at least, to be finished.

Stewart: It really is. We worked hard on a lot of things, and it’s nice to see them see the light of day.

GamesBeat: I remember your talk from Magic Leap’s debut. It involved a lot of goats. Are there goats in this thing?

Stephenson: The baby goats project was the brainchild of Karen Laur, who was part of the creative team that came up with this world. But baby goats was an unrelated idea. Why not populate your living room with goats that run around and jump up on tables and are responsive to the environment? In order to build that application, we had to do a lot of fundamental engineering work around — we built a debugger. We built some other utilities for analyzing a room, breaking down the geometry of it, creating pathways. All of that was fundamental building block stuff that needed to be done.

The direction that project took became a demonstration code project. We made that application work at a basic level, and we released several tiers of technology, beginning with the debugger and building on top of that, so that other developers who wanted to create their own applications on the Magic Leap platform could use that sample code as a template to work from. We then used that underlying tool set to build the location-based experience work that we did for the New Found Land IP.

But you may remember, if you saw that presentation that I gave–that would have been October of 2018. I had flown directly there from New York, where we had just finished up the ARG and the activities at New York Comic-Con. If you were following the timeline, the release of the videos and so on, there’s a direct connection to that, and a little statement I made during that presentation. I showed a blurry picture of someone leaving the Magic Leap facility, carrying a lunchbox, a beat-up lunchbox full of electronics. That’s the PHILTR, the fictional device that figures into the story. That was the moment, according to our storyline, when one of our engineers went on the lam and ran off to Idaho with the PHILTR in the back of his car and has never been seen again.

Above: Rony Abovitz is the founder of Sun and Thunder and former CEO of Magic Leap.

Image Credit: Sun and Thunder

GamesBeat: It sounds like you made great progress in figuring out how to tell stories in AR.

Stephenson: I’d like to think so, yeah. We did a lot of thinking about what storytelling means in that kind of environment. How to build an interactive experience that would be interactive, and yet reveal things about the world, let’s say.

Stewart: One thing that’s tricky when you’re making this stuff– if you went to film school, they spend years teaching you to have the exact wrong instincts for making AR. You always have control. The three most important people on a movie are all there to do one thing, which is control what you see at every second. The director, the director of photography, and the editor, that’s all they do. When you walk into AR and VR you’re in a different world. The audience can just look at something else.

Rony was talking to Neal one day and he said, “It’s weird. If you think about VR, everyone knows how to make VR, because you just set things in Middle-Earth and you’re in Middle-Earth. But with our technology, it has to be amazing and beautiful and wondrous, and yet it’s happening in your living room. How do you tell that story?” Which is the point where Neal said, “I think I know a guy.” Which is how I came to be at Magic Leap in the first place.

ARGs are a very natural jumping-off point for creating in AR, because in both cases so much is about letting the audience discover, explore, speculate, and find the story and assemble it for themselves. You’re not going to be able to just nail them to a chair and tell them to stare at a screen while you do your thing. It has to be much less of a one way street, and much more of a dance. You say, “I have some steps. If you follow me and listen to the music, we can have some fun together.”

GamesBeat: You’re convincing the user that there’s magic in the real world.

Stewart: That’s always been my goal. When I was young I desperately wanted to go to Middle-Earth, and I couldn’t find a way to do it. I’ve been trying to backwards-engineer that ever since.

Stephenson: There’s another thread that became interesting to us over the course of the project, which is just non-interactive entertainment within AR. Epic and the Unreal Engine have been getting heavily involved in moviemaking. They’ve beefed up the quality of the tool set within Unreal Engine that’s used for sequential entertainment. You create a world and then you program certain movements and events that happen there.

Normally you see that all through a camera, which can also be programmed. But you don’t have to. You can also use Sequencer to make three-dimensional movies, as it were, that a viewer wearing AR or VR equipment could walk around and see from different angles in three dimensions. We also got pretty deep into working with that, because it’s a way you can just tell a story using game engine technology without the added complexity of making it fully interactive.

Above: Johnny Delta comic book

Image Credit: Sean Stewart

GamesBeat: Just to be clear, when you say there’s no ARG, there’s really no ARG?

Stewart: There’s no ARG. There’s no current ARG. Good question to ask, but no.

GamesBeat: There’s so much fiction here that’s mixed in with the real world, I have to make sure.

Stewart: I understand! Needless to say, it’s been a strange couple of years for those of us who started in the ARG space.

Stephenson: Sean knows what it is to launch an ARG for real and then have to deal with the onslaught of change and heavy interaction and hasty improvisation that goes with that. He’s not one who would idly touch one of those things off.

GamesBeat: It could be that you just don’t know. Maybe someone from the alternate universe took over Magic Leap, and that’s why you got shut down.

Stephenson: It is noteworthy that just as we were getting going on this project in a serious way, suddenly, boom, it was all gone.

Stewart: The website is now an Indonesian travel blog. Do you think that just happens?

GamesBeat: They got to Rony.

Stephenson: He’s a true believer. He’s one of us. He’s been very supportive.

Stewart: Or so we think!

Source Link