Gaming’s Biggest Week Gives Clues to Serious Technology’s Future

This week is one of the biggest of the year for those like me who enjoy playing video games. That’s because it’s time for the 2019 E3 Entertainment Expo, one of the largest trade shows for games and gamers in the world. Most of the major video game publishers, and a few of the development studios, will be gathering in Los Angeles to demonstrate all the best new games and gaming hardware that they plan to release later this year. And while I am surely excited about everything from new role-playing titles to the remake of the classic SpongeBob SquarePants Battle for Bikini Bottom game, at least some of the innovations being shown will have more serious applications that may one day make it into business or even government.

Those of you who regularly read my Nextgov column know that I try and keep up on official government gamification projects, which have waxed and waned as a priority over the past 20 years. Right now, official efforts in this area seem to be centered on simulations and training, though even there, much of the effort is taking place at the state and local level. A good example of this was on display last year when the Serious Play Conference, which examines how to take game technology and modify it for more serious uses, swung through D.C. There were many examples on display showing how simulations could help not only for training, but to test out various responses to dangerous scenarios in an effort to help plan how to best handle them.

“Simulations are good for the discovery of the previously unknown,” said Justin Legary, who previously worked as the tech coordinator for the Federal Emergency Management Agency before moving to the private sector. “This is especially true in recent years, when, with the help of computers you can perform hundreds or even thousands of run-throughs instead of just a dozen or so. This allows you to see patterns that were not possible to see before.”

Simulations are interesting, but I don’t think they are the future of either gaming or most of the really impressive serious game technologies of the future. Sure they will play a role, but the deep innovation is going to be elsewhere. Some clues as to where those innovations will be coming from are being revealed at E3. And I think it will come down to two main technologies, virtual reality and augmented reality.

Feeling Around New Virtual Realities

Everyone is probably familiar with the concept of virtual reality (VR) these days. You strap on a large pair of glasses and they shut out the rest of the world, instead rendering a totally new one. Since humans get about eighty percent of their sensory input from their vision, its surprisingly easy to trick your brain into thinking that what you are seeing is, in fact, reality.

I can attest to this firsthand. While getting a demo of a pending virtual reality game, I was walking through the bar on a space station, which looked something like the Mos Eisley cantina from Star Wars. Now I know for a fact that I was not on Tatooine, but since my mind was seeing it, and my ears were backing that up, it became quite easy for me to accept that new reality after a couple minutes. That is until I decided to play it cool by leaning up against a nearby post so that I could quietly survey the bar and look around for my smuggler contact—and promptly fell on my butt. The post, of course, did not really exist, so I just went right down to the floor.

That embarrassing moment happened using today’s technology, meaning the bar did not really look much like reality, just enough to trick a dummy like me. To render things in VR these days that look just like real life requires a lot of computing hardware. It’s possible, but not really practical. That’s bad for gaming, but not too much of a big deal in the world of government or other serious simulations. So long as a firefighter knows what fire and smoke looks like in a training module, they will be able to learn how to react to it. So asking for better graphics, while helpful, is not what is holding VR back from serious activities.

No, the biggest problem with VR is how humans must interface with their virtual worlds. Check out this screenshot from the SuperChem VR title, which was developed by Schell Games with a grant from the Education Department. What looks off about their cool virtual lab?

Hint: It’s not the lab itself.

The player’s hands are the problem. He’s not really able to interact directly with the environment, but must instead use a pair of clunky grabber sticks that the computer can track. Almost all VRs today are like this. Games can get away with it if you spend the whole time lugging around a gun, but for things like NASA training programs, the extra interface between your hands and the world is a big impediment.

That could change with a new invention from CTRL-labs that will let users put their real hands into a VR. The prototype device works by strapping around your wrist. As you move your hand, your muscles contract in a certain way, and the device knows how that translates into the position of your hand. It’s completely non-invasive, and feels like a large wristwatch. Yet it allows for real interaction with objects inside a VR with no camera, joystick or other buttons required.

While companies struggle to improve the look of VR worlds, it’s just as important to also improve the way we interact with them. This more than anything else will help the technology make the leap from fun and games to more serious applications. That’s according to CTRL-labs’ Director of Research and Development Adam Berenzweig, who I was lucky enough to interview. And the work to put more human parts into virtual worlds won’t stop at just their hands, many more input innovations are right around the corner.

Better than Reality with Augmented Reality

For gamers, augmented reality (AR) is actually kind of a big step down from VR. Instead of a space station or underwater metropolis, the background for AR is whatever happens to be around in real life. But for government training and simulations, AR is probably much more useful, even in its current state that is still very much in development.

In AR, people still generally put on goggles, though they are clear. So you are still aware of your surroundings and won’t accidentally lean up against any phantom poles or sit in a chair that isn’t really there. The computer takes your actual surroundings and then adds images to the screen to augment reality, without pulling a person completely into a virtual world. AR can even be achieved using a smartphone’s camera, though the immersion level is obviously much lower.

Examples of AR are on display at E3 this year, including the huge, 6,000-square-foot Unreal Garden exhibit. The idea there is that developer Onedome has built a pretty fantastic setting, and then users can get really tripped out by having even more crazy things pop into their reality through a set of goggles.

In the picture of the garden, the little hobbit-looking house is real. But the floating jellyfish creatures are augmented by the program to heighten the experience. It’s incredibly cool.

While government probably isn’t interested in rendering flying jellyfish, the possibility of using AR for training is practically unlimited. Imagine loading up the specifications and pictures of every part on an F-22 Raptor and then equipping an Air Force mechanic with AR goggles. They could visually inspect each part versus what it’s supposed to look like, and immediately spot any deviations. Then a video could show exactly how to service or fix those same parts, all without tying up their hands.

And then there are combat type training applications. The military, as well as state and local law enforcement groups, could easily use AR to train their troops, subjecting them to almost any kind of situation or threat, all within a real world environment. So it will seem very real to the trainees, without the need to render a virtual world.

Although AR is generally less developed than VR, the potential is there to make an even greater impact on serious applications. And while VR still needs a lot of development, AR could potentially start making a difference right now. Government should seriously rejuvenate its gamification efforts. The payoff with things like AR and VR could be almost as amazing at the technology itself.

John Breeden II is an award-winning journalist and reviewer with over 20 years of experience covering technology. He is the CEO of the Tech Writers Bureau, a group that creates technological thought leadership content for organizations of all sizes. Twitter: @LabGuys