SOEmote implants your facial expressions, movement, and voice onto your EQ2 avatar

The future is here and it's full of Ratonga.

The future is here and it's full of Ratonga. SOE is adding new facial recognition tech to EverQuest 2 that lets the game track your movement and facial expressions and replicate them on your avatar in real-time. Voice chat is also built into the feature so that your character will animate naturally while you talk.

We have a full demo video of the tech below, and the inside scoop on how it works and why the devs think MMO players will love it from EQ2's Director of Development David Georgeson and the CEO of the company that developed the technology, Robert Gehorsam.

PCG: Aside from it being cool future-sci-fi-world technology, why is it important to get this tech into MMOs??

Dave Georgeson: Any game that calls itself a roleplaying game absolutely needs this feature. For other games, it's just a really good idea that your players will want.

Pick a character. Pick a class. And then be that character instead of just being yourself. That's what a roleplaying game is supposed to be.

You're you for 24 hours a day, 365 days a year. When you play a MMORPG, you get to be someone else. Someone bigger than life that can be heroic or evil, passionate or savage, but someone that lets you stretch out beyond the normal RL envelope of life while simultaneously becoming legendary, fighting dragons and demons and saving the world. SOEmote lets you do that in a fashion that you've never experienced before.

PCG: Can you explain exactly what the software is tracking?

Robert Gehorsam: Within EverQuest 2, the Live Driver software essentially performs detailed measurements of the player's facial expression on every image captured from their webcam. Today's webcams typically operate at 30 frames per second. Our software tracks 64 points and makes around 100 expression measurements per frame, for over 5,000 measurements per second.

PCG: As a player, a lot of my time is spent looking at the back of my character's head, and other characters can be far away from me. How useful will this really be, especially in combat?

DG: When we first started creating this feature, our assumption was that you wouldn't really be able to see the facial distinctions at any distance, so we figured we would need to add a pop-up window so that a player could essentially create an in-game version of Skype with another player or create Brady Bunch-style windows for group interface.

But what we found after we got it into the game was surprising. Humans are very good at discerning even the tiniest differences in facial expression at much greater distances than you can imagine. Even when a face is tiny at distance, your eye is still noticing important elements, like an eyebrow being raised, a grinning mouth, or even eyes squinting in concern. The more we experimented, the more we realized we didn't need a close-up window at all.

But is SOEmote designed to be useful in combat? Not really. Combat already has a million things going on that you need to watch, and checking out the nuances on a fellow player's face while you're supposed to be keeping that player alive is not what we were shooting for here.

SOEmote's “purpose” is simply to let players communicate more effectively, to immerse into the world more deeply, and to allow roleplaying to return to our roleplaying games.

PCG: So it's being designed primarily as a roleplaying tool?

DG: SOEmote is, of course, a fantastic roleplaying tool. That method of gaming got pretty seriously beat down when we added VOIP to MMORPGs. Of course, it's far easier to voice chat than to type in a text pane, but that ease-of-use improvement had a high consequence. Once players revealed their real voices to each other, it suddenly started feeling awkward to pretend to be a big gruff troll or a high elf princess… and roleplay took a serious nosedive.

But now, with SOEmote, you can make your character's face come to life, and also filter your voice to sound like your actual character (thus restoring anonymity and enabling roleplay again). It's amazing how much life suddenly springs into your character when those two elements combine to make your character become believable.

Our objective with SOEmote was to enhance people's ability to immerse within the game. Yes, that definitely makes roleplaying a lot easier, but it's also great for anyone that just wants to hang out and talk to their friends. It really has to be experienced to fully understand the impact.

PCG: Will players be given the option to opt out or pause their input without unplugging their webcam?

DG: Absolutely. SOEmote is really easy to use. Plug in your webcam, open up the SOEmote panel, hit Calibrate, select a voice font, and ta da you're done.

Then when you're playing, and you want to stop the camera feed, you just hit a hotkey and the face data and voice stops immediately. At that point, your character's face returns to its normal neutral pose. Just use the hotkey again and you're back on SOEmote and transmitting.

It should be noted that your actual image never gets transmitted at all. All the webcam is doing is watching key spots on your face and transmitting that positional data to the game. Your facial images are never actually sent or stored anywhere.

RG: One advantage of the technology that powers SOEmote is that it provides anonymity to the players. Images are analyzed in real-time. The resulting data is only used to provide animation values and then it's discarded.

PCG: Aside from roleplaying, how else do you expect players to use this tech in-game?

DG: The feature wasn't designed to be used in any specific way. This is one of those emergent gameplay features. We're enabling it with some basic ideas of how it will be used, but with the intent to sit back and watch how players use it.

But what can we imagine? Externally to the game, Machinima is pretty obvious. We already have YouTube, Facebook, and Twitter support within the game. I can picture some pretty amusing stuff happening on that front. Player creativity is boundless and anything we can imagine here will be dwarfed by what the players actually do.

But within the game, whether or not you're interested in roleplaying, the ease of communication when using SOEmote is tremendous. Anything that breaks down communication barriers between players is fantastic for MMOs, and something that also bolsters immersion and roleplay in this manner is tremendous.

We did support a couple of specific behaviors. For example, we've set things up so that guild leaders can broadcast a live feed via a special guild window. Sort of like a live TV broadcast. The leader opens a window, the guild members choose whether to receive the window or not, and the leader can then broadcast via SOEmote to the entire guild at once. Neat, eh?

We're really just scratching the surface with this initial release of SOEmote. We're working with a fantastic partner, Image Metrics, that has been developing and using this technology for filmmaking now for many years. They are indisputably the best people in this space and they're continuing to develop new ways to use the webcam interface. Like hand tracking, or recording your own full-body emotes.

Immersion is a fantastic thing and we're pushing it hard here at SOE.

PCG: Has this technology been used in anything I might recognize?

RG: Prior to Live Driver—the technology used in EQ2—our professional facial animation technology, Faceware... was used in more than 40 AAA gaming titles, including Red Dead Redemption, GTA IV, Assassin's Creed, Max Payne and more.

PCG: I imagine there were some amusing bugs during testing.

DG: When we first implemented the SDK from Image Metrics, the software didn't clamp down the feeds if you turned your head too far left or right. It got into a very strange math state and your head suddenly started flipping out like a scene out of Lawnmower Man or Max Headroom. We had a lot of fun with it before finally fixing it.

Also, watching someone use SOEmote for the very first time is always hysterical. Once people get used to being anonymous in-game again, they start really stretching their faces and voices to push the limits of the feature, and that's funny stuff.

PCG: Have the devs found any particularly creative or surprising uses for the tech?

DG: One of our animators immediately set up all kinds of hotbar shortcuts for various character emotes and spent far too much time singing via SOEmote and being a tavern dancer. It was hysterical. You really don't need to see a Ratonga dancing to Britney Spears. (Okay, you do. But just once.)

PCG: So it can handle Britney Spears, but how subtle of changes in facial movement can the technology actually pick up on and display?

RG: It's not just a simple face tracker: [it can] track highly-accurate facial expressions in real-time, including subtle micro-expressions, which allows the player to see the intent behind the facial gesture. Not only does it capture broad movement like head pose, smiles, and blinks, but also the intent of more nuanced expressions: a sly smile vs. a snarl, surprise vs. an arched eyebrow. It also captures asymmetrical movements—something that's important in gameplay.

PCG: With tech like this and constant voice chat, do you see text chat eventually disappearing from MMOs?

DG: Absolutely not. Text chat is going to be around for a long time and I see no real reason for it to go away. Area chats, LFG channels, and brokering are all elements better suited to chat. Also, some people just like to carefully compose their thoughts before sending them and text is better suited for that.

What proportion of text chat will this replace? The players already use VOIP now. SOEmote will certainly encourage some new folks to start using VOIP so they can roleplay, but I don't expect the balance of conversation to change immediately. Stuff like this takes time to ripple through communities and they'll have to decide how they want to use it. But the potential is tremendous.

PCG: Do you think this tech could or should be used in all games, or is it peculiarly useful in EQ2 for some reason?

DG: My personal opinion? All games should have this. You've got to try this. Once you get used to other people's faces moving realistically and intuitively, characters in other games start to feel really plastic and mannequin-like.

Human communication is all about face and body language. You already know this because you currently use video feeds whenever possible to communicate over the net, either through YouTube, Skype or Facetime, rather than relying on just VOIP or text.

SOEmote allows you to have that level of communication within a virtual world. Now that this is possible, why would you want anything less?