LEGO and Augmented Reality

There’s no doubt in my mind that augmented reality is going to be amazing and that people will build incredible new experiences with Apple’s ARKit, version 2 of which was announced earlier this week at the company’s annual World Wide Developers Conference. But in order for that reality to come into focus, developers—and designers—are going to need to bring greater thoughtfulness and creativity to the new medium, or at least more than was on display during the LEGO demonstration portion of the keynote.

If you missed it, you can watch the video, below. In it, LEGO director of innovation Martin Sanders and a colleague use iPads trained on a real, physical LEGO model to unlock a layer of augmented reality. The real time visual coordination of the physical and the virtual is impressive. Sanders claims that the combination of the two “really opens up those creative play possibilities.”

I found it to be a surprisingly thoughtless take on how kids actually play with LEGO toys.

My kids have been obsessed with LEGOs for about a year now, and so I find myself, somewhat unexpectedly, with a lot of opinions about the experience of playing with bricks. We’ve built a lot of kits together, roughly about as complex as the one shown in this video, and I can easily imagine them being enthusiastic if those kits could be combined with what is essentially a video game layer. That’s not unique to them, of course; pretty much any five year old in Western society is going to be wildly receptive to any video game opportunity.

But that’s at cross purposes with what I, as a parent, find most valuable about LEGO bricks: their tactile, physical nature. I fully buy into all the clichés of the benefits of LEGO play: they build fine motor skills; encourage problem solving; enhance coordination; stoke the imagination; and even teach the value of cleaning up after yourself (that last one is not to be underestimated). And in a major bonus, they do all of this without batteries, keyboards or screens. For our family, LEGOs are great because there’s no tech involved. The fact that they are not video games, that they engage our kids in a wholly different way from video games is a valuable feature, not a shortcoming.

Even setting aside my mildly Luddite attitude towards digital toys, what struck me about yesterday’s demo was that it fundamentally and carelessly subverts the purpose of LEGOs. Presumably, once a set is completed, it must remain more or less intact in order for the AR component to work. Sure, you could probably change up a decent portion of the parts and the software could be smart enough to account for the change, but it seems logical that if the kit were to be disassembled too much, the VR experience would stop working entirely.

The problem with that approach is that, at least in our household, the completion of a LEGO kit is just one stage in its useful life, so to speak. Even incredibly complex models that took hours or days to complete will come apart eventually, either through the natural disassembly that happens when they’re stored in toy bins or through purposeful dismantling. For us, no LEGO brick ever has a truly permanent use.

In fact, what happens to bricks after they’re combined to look like what you see on the box is, for me, much, much more interesting. My kids and I regularly sit down and build fantastical new creations that pull parts from countless other kits, whether they’re wheels from a police car, arches from a building, a rowboat from a picnic set, or maybe weirdly organic shapes from a LEGO dragon. Here’s an example of one that we’ve been working on for the past two weeks:

Improvised LEGO Construction

Anything and everything goes into these odd assemblages; they have no plan and no purpose. That’s what makes them so fun; they’re free-associative improvisations with no real limits or constraints other than that they need to be stable enough to stand on their own. And even then, if they fall apart, that’s fine too; we just remove what didn’t work and then we add something else. There’s no wrong answer to the question “What do we add next?”

This freeform method of play is what truly unlocked LEGOs for my kids and me. When they first started getting obsessed with them, I was somewhat cool to the process of following the extensive instruction booklets that are necessary to assemble the sets—to me, they just seemed like preparation for a life of putting together IKEA furniture. But once we removed the rules and the sense that anything we built was ever meant to stay that way, it became much, much more interesting for everyone. Now we collaborate on these creations together, and anything the kids want to add is just as interesting as anything I contribute—usually more so.

That kind of play seems incompatible with what the LEGO team presented at WWDC. Their vision of combining bricks and augmented reality changes the goal from building for the joy of it to building in order to unlock a video game. The assembly of a LEGO kit becomes just a preliminary stage in spending more time looking at screens. And as any parent will attest, the allure of screens for kids is so potent that this new take on play effectively limits the usefulness of the physical toys. Disassembling a LEGO model, reusing its pieces for other creations—these natural behaviors are inhibited when AR is introduced in this way, because they would cut off that intoxicating gaming layer. There’s not a kid out there who would be willing to take apart something that allows him or her to spend more time on an iPad.

To be fair, early applications of new technologies are often shallow interpretations of the true potential of the medium at hand. As we become more acquainted with what AR can do, it will become more apparent both how AR can be used more effectively as well as how AR should be used. Augmented reality has the potential to enable true innovation, but it would be a shame if that potential is mostly harnessed to subvert what works so well without it, as it seems to do in this case. For designers and developers, when we think about what we can make with technologies like this, it’s perhaps more important than ever for us to think about what is good for our users—is it really in the best interests of kids to use LEGOs just to play video games?

I’m certainly not arguing that makers of physical toys shouldn’t be investigating how AR can complement their products. It does seem logical that there will be something pretty interesting to come out of combining LEGO bricks and AR—maybe an app that you point at a pile of random LEGO parts that then shows you what new creation can be made from those pieces? Or an app that lets you point at any object and then generates instructions on how to build a LEGO version of it? Maybe an app that lets you identify two or more kits that you own and shows you how they might be combined? Concepts like these are admittedly more complex to execute than simply adding a video game layer to an existing kit, but if augmented reality is to be as truly game changing as it’s been advertised, it probably won’t be enough to settle for simple concepts like what was shown by the LEGO team at WWDC. This new immersive future is going to require us not just to build more ambitious products, but to be more thoughtful about them, too.