GDC Classic: Multiple character interaction in Petz games

In this classic presentation from CGDC 1998, Adam Frank and Andrew Stern explain what working on their seminal virtual pets franchise has taught them about designing believable characters.

In this classic presentation from CGDC 1998, Adam Frank and Andrew Stern explain what working on their seminal virtual pets franchise has taught them about designing believable characters. Presenters were required to submit a formal paper at the time, and we are reprinting the original paper as it was written.

Multiple Character Interaction
Between Believable Characters

PF.Magic
501 2nd St, Suite 400
San Francisco, CA 94107
+1 415 495 0400
http://www.pfmagic.com
http://www.netcom.com/~apstern

Introduction

Computer games have always had characters. In a typical game the player controls the “main character” to fight or compete against one or more computer-controlled “non-player characters”. The game’s “artificial intelligence” (AI) controls the behavior of the non-player characters in an effort to out-maneuver, out-shoot or out-smart the player.

But even to this day, with few exceptions, computer characters are really no more than shallow cardboard-cutouts. While they are beginning to look more realistic, they still act in very limited and often predictable ways, with almost no variety in their behavior. They tend to do and say the same things over and over no matter how hard the player tries to get them to react differently.

Fortunately, as the power of the personal computer grows, it is becoming more and more possible to create truly lifelike and believable autonomous characters. In this paper we first propose what lifelike, believable characters should be, and design principles involved in making them. The first multiple character interaction we discuss is between the user and a single lifelike computer character. Next we discuss what should happen when multiple lifelike computer characters interact with each other, and what new kinds of interactive experiences for the user can result from this.

We use the Virtual Petz we developed at PF. Magic to demonstrate these key points. The original Dogz , initially released in 1995, was the world’s first virtual pet, followed soon by Catz , Oddballz and the new Dogz II and Catz II . Virtual Petz are autonomous real-time 3D characters that live on your PC computer desktop, grow up over time, and strive to be the user's friends and companions.

What are Lifelike, Believable Characters?

These days most people think the way to make a computer character more “real” is to make it look more photorealistic, and to give it a better AI. Since almost all interactive entertainment software products are games, the typical AI is designed to figure out the best move to make, such as the next punch or kick move, where to run to and shoot from, and so on. But figuring out the best next move is an incomplete definition of AI. When people talk about wanting great AI, or great “artificial life” for that matter, what they really want is the experience of interacting with something truly alive.

When would a player believe a computer character is truly alive? What constitutes a convincing, interactive “illusion of life”? It seems that the best way to define what truly alive means is to study real living “characters” – that is, animals and people.

- Real living things are very reactive and interactive. Given this, when users physically interact with a lifelike computer character in its environment, the character should be aware of what is happening and react immediately. If a character reacts in a way that makes no sense, or doesn’t react at all, then the illusion of life is broken, and the character cannot be considered believable. Also, real living things don’t always react the exact same way each time! A believable character should not appear to be repeating itself verbatim every time the user interacts with it. Along the same lines, if the user is given little or no way to interact with the character in the first place, the whole interface itself does not support believability.

- Real living things are able to communicate. Most animals, and people especially, have several ways to communicate with each other. These include gestures, actions, body language, sounds, and spoken or written language. To be truly believable, users and computer characters should be able communicate with each other in a natural, lifelike way. If a user comes face-to-face with on-screen human character, they really should be able to speak to communicate, not type.

- Real living things are very expressive and able to show a broad range of emotions, desires and personalities. A believable character wouldn’t just fight or shoot, it would have a broad base of behavior – the ability to eat, sleep, play, chase, explore, groom, build, and so on. Lifelike characters should be emotional as well. Depending on what is happening in their lives they would act happy, sad, angry, afraid, jealous, and so on. They may get energetic or tired, hungry or full, sick or healthy, and may tend to be grumpy, hyper, sweet, or finicky. All of these emotions, moods and personalities must be expressed by how the computer character behaves, animates and sounds. Even if these things are being kept track of in the character’s mind, if they’re not expressed, they’re effectively not there.

- Real living things are able to learn, adapt and change over time. Believable characters should be able to pay attention to what is going on in their environment and learn from it. They should have attitudes and feelings towards objects and other characters, and act appropriately upon these attitudes. They should form relationships with the other characters in the environment, including the user. As a believable character grows and ages over time, its attitudes and relationships will change, and it will gain experience and knowledge.

If you follow what’s happening these days in academic media research laboratories, the concept of believability is now a key design goal for many synthetic character research projects, such as the Oz Project at CMU (1) , Improv at NYU (2) , Synthetic Characters at MIT Media Lab (3) , the Virtual Theater Project at Stanford (4) , and so on. Refer to my personal web page for a complete listing (5) .

Examples of Believable Characters: Virtual Petz

Clearly, making truly lifelike believable characters is a difficult problem. At PF.Magic we created the Virtual Petz characters using the above criteria, incorporating as much of each component of believability as we possibly could. As far as we can tell, the Petz are the first consumer software products to address all of these criteria.

The most direct way the user can interact with the Petz is through petting. Using a mouse the user moves a hand-shaped cursor to pet, scratch and stroke the character. The pet immediately reacts in a variety of ways depending on where its body is being petted, how fast, and how the pet feels at the time. By right-clicking the user can pick up and carry a pet. We call this direct interaction. The fact that you can touch the character is a very effective way of building an intimate relationship and creating an illusion of life.

The toys and objects in the pet's environment have direct object-like interaction for both the user and the Petz . By picking up and using a toy, the user can initiate play. For example, throwing a ball may initiate a game of fetch, or holding a tugtoy in front of a pet may initiate a game of tug-of-war. Similarly a pet can get its own toy and bring it to the user to initiate play. This cooperative decision-making helps build the owner-pet relationship.

We have created a variety of personalities - playful terriers, grumpy bulldogs, hyper Chihuahuas, finicky Siamese cats, lazy Persian cats, aggressive hunter cats, timid scaredy cats and so on. Additionally each pet has its own individual likes and dislikes, spots and body coloration, and personality quirks. Users get to play with a pet and see if they like them before deciding to adopt. Once adopted, the user gives them a name. This individual variation allows the user to develop a unique relationship with a particular pet. Every owner-pet relationship is different.

Petz communicate with the user in a variety of ways. Emotion can be expressed through different facial expressions (eyebrows, mouth, ears), styles of movement and body language (sad walks, happy trots, various postures, a variety of tail motions), and sounds (excited playful barks and meows, sad whines and whimpers, yelps of pain, etc.). They get fat when they eat too much and get skinny when hungry.

Petz communicate their intentions and desires through their actions. We have created a broad base of behavior for the Petz -- they can eat, sleep, play, attack, groom, hiss, explore and so on. When Petz want attention they may start barking or meowing. If hungry they may go searching for food, hunt mice, or start begging. When upset or scared they may run and hide, act aggressively, or cower and shiver.

The user is given a few specific hand gestures that Petz may respond to. By double-clicking in an empty space on the screen the hand cursor animates a "come here" motion and plays a whistling sound. Petz tend to stop what they're doing and come to the user's cursor. Users can also pick up and wiggle food treats in different directions to train their Petz to jump, sit, rollover, somersault, and so on. Additionally, users can reward or discipline any behavior. Methods of rewarding include a giving treats, feeding, and petting, while discipline is achieved by squirting a pet with a water spray bottle. In this way users can modify a pet's overall behavior through positive and negative reinforcement. Additionally, the personalities of each pet can change under extreme circumstances. For instance if any pet is underfed it will begin hunting and acting aggressively. If a pet is overfed, it will become fat and lazy. So depending on how their user treats them, their behavior can slide along a broad spectrum of personalities.

A User-Perception Based Approach

It is important to note that the only way the user can understand what the Petz are feeling is to interpret their actions and physical cues, in the same way an audience interprets an actor's performance. We do not display bar graphs or text messages describing the pet's internal variables, biorhythms or emotional state. By forcing a natural interpretation of the pet's behavior, we don't break the illusion of a relationship with something alive.

Again, “real” AI or A-Life should be defined as the illusion of life. In entertainment software, all that matters is the strength of this illusion. It does not matter what internal techniques are used or how sophisticated the programming is. Lifelike believable characters are ultimately perceived and judged through human eyes. All that matters is what you see and believe on-screen. This should be the standard by which any believable characters are held to – does the audience believe that the characters are alive. Simply making a computer character look photorealistic won’t work – without the interactivity, communication, expressiveness and adaptability, it’s just a mannequin.

Design Principles for Making Lifelike Believable Characters

Certainly there are many ways to go about making lifelike computer characters. In the process of working on Virtual Petz we have come up with a few design principles that have worked well for us.

- Choose characters you can successfully implement. For Virtual Petz, choosing dogs and cats as our characters helps us in a few ways. People understand what pets are and essentially know how to interact with them. This is critical for producing believable characters. Pets are also a good choice because they are expected to behave in ways that we can successfully implement. They have relatively simple gestures, language and cognitive abilities. Also, animated dogs and cats have well established cartoon archetypes. We base the look and behavior of the Petz on these archetypes.

- Frame the experience in a looser, more character-centric way. Instead of the typical goal-oriented computer game, the Virtual Petz experience is non-goal-oriented. We allow users to explore the characters and their toys in any order they like. This freedom allows users to socialize with their Petz in their own way and at their own pace. This also encourages users to come up with their own interpretation of their pet's feelings and thoughts. Often the scope of their interpretations far exceeds what we originally planned for.

Note that we have focused almost all of our energy on creating rich characters and not on creating a sophisticated environment. The Petz come to the user’s environment – the desktop – not the other way around. We only built the objects and toys that were needed to support the interaction with the characters. By doing this we are making the Petz the stars of the show.

- Design your animation to be as dynamic and responsive as possible. A high degree of interactivity and lifelikeness truly requires the use of real-time 3D animation. Pre-rendered animation, however beautiful it may be, cannot be as responsive, dynamic, or varied as required for lifelikeness and believability. One should choose to err on the side of frame-rate versus a photorealistic image that interacts poorly.

The Petz products use a network of basic 3D body animation which has a high resolution of branch points. For example, from any basic pose such as standing or sitting, the pet can transition to any other pose. As the pet's general body animation moves across this network, additional animation is layered on top. This allows the pet to look in all directions no matter what its body doing. Also we layer breathing, blinking, meowing, head shaking and even character-specific posture. In fact the look of the each breed and their ages are layers. The subsequent number of permutations is enormous and therefore passes a critical threshold in human perception. This method greatly enhances our ability to create an illusion of life.

- Don’t over-engineer the behavior logic. The Petz behavior is controlled by a variety of mechanisms that operate simultaneously, each with varying complexity. There are small mechanisms to control blinking and breathing, biorhythm mechanisms to keep track of things like energy and hunger, and a variety of cognitive mechanisms to keep track of multiple goals, different plans to execute each goal, multiple emotions towards other characters and objects, attitudes towards other characters and objects, and personality-specific quirks. In general we’ve been able to reduce complexity and still retain a high-degree of lifelikeness and freshness with some relatively simple techniques. To achieve a real, unpredictable, non-repetitive quality in these mechanisms, we often use constrained randomness. For example, there are rules for each individual pet that determine which toys it most likes to play with, but there is some randomness in that decision. This results in the possibility that a pet might play with any toy, not just the toys it likes the most. A lot of development time is spent tweaking the constraints on the randomness to achieve a good balance between consistent personality and unpredictability. Again we are focusing on the user's perception of the character and being careful to not create a behavior system that models more than can be expressed.

Interaction Between Multiple Lifelike Computer Characters

Now that we’ve talked about what believable characters are, how should two lifelike computer characters interact with each other? Again let’s look how real living things behave. Animals and people perceive and react to one another in all sorts of ways -- they look each other over; approach and investigate one another; communicate using gesture, body language, sounds or speech; directly contact one another by touching, holding, fighting, hugging, mating, and so on. A believable character should be able to do all of these things.

On a conceptual level, the two are forming a relationship with each other. Believable characters have attitudes and feelings about one another – how much they like each other, how much they know about each other, what they can gain from each other. This relationship evolves over time as they interact and events happen. If two characters help and support each other, they should become friends; likewise if they hurt or abuse another, they should become enemies. The characters’ individual personalities should influence the ways and rate at which the relationship can change – a very timid character may quickly distrust another at the slightest provocation, whereas a happy-go-lucky character would need to be severely abused to begin distrusting others.

Again we’ll use Petz as an example. In our newest products, Dogz II and Catz II , where two Petz can interact with the user and each other at the same time, we divided different levels of multiple character interaction into three categories.

- Looking but not touching. A lot of meaningful interaction can happen without any physical contact between characters. For the Petz this included looking at each other, investigating each other, following each other, barking or meowing at each other, ignoring or snubbing each other, and stand-offs like growling and hissing at each other. A lot of information about how two characters feel about one another can be expressed this way. These interactions were easiest to implement because they didn’t require perfect timing synchronization between the two characters.

- Playing with a common object. This is similar to looking-but-not-touching, but now includes a common object which the interaction is focused around. For the Petz this included things like playing fetch together, chasing mice together, playing frisbee together, sharing food and water bowls, bringing each other toys and food, spraying each other with the water spray bottle, and so on. Again this doesn’t require perfect synchronization between the characters because they never physically contact each other.

- Direct contact. This includes wrestling, fighting, tug-of-war, grooming each other, carrying each other, bumping into each other, licking noses, and mounting each other. These behaviors required physically aligning the bodies of two characters in space and time so they contacted each other properly, and then drawing their bodies so they accurately meshed in 3D. These types of behaviors are perhaps the most high-impact, because they are so lifelike and intimate, but also the most time-consuming and hardest to implement.

Developing Relationships

Using the above multiple character interactions, we implemented all sorts of different behaviors to allow Petz to express their relationships between each other and the user. Several different categories of relationships can be identified.

- Buddy relationships. When two Petz really like each other, they play a lot together, bring each other food and toys, groom each other, follow each other, lick noses, sleep side-by-side, and so on. Buddies act happy when they first see each other and act sad when they part. They will get angry at the user if the user abuses their buddy. If a pet considers the user their buddy, they will be affectionately lick the user’s hand cursor, always come when called, obey commands and gestures, etc.

- Enemy relationships. When two Petz hate each other, they growl and hiss at each other, wrestle, fight, steal toys from each other, scare each other, spray each other with the spray bottle, and so on. Enemies are happy when the user disciplines the other, and are happy when the other goes away. One get jealous if the other receives attention and affection from the user. If a pet considers the user their enemy, they will act naughty, ignore the user, bark angrily, and even eventually runaway.

- Nurturing relationships. These are typical between adult Petz and younger Petz. Adult Petz will comfort and protect younger Petz if they get scared and carry them to a pillow to sleep. They may bring them food to eat or toys to play with. Younger Petz may follow the adults around and attempt to copy their behavior. Adult Petz may even try to teach younger Petz how to do tricks.

- Neutral relationships. Sometimes two Petz don’t have strong feelings for each other one way or another, and they basically play independently of each other. However the user can often encourage two neutral Petz to play together, or taunt them to get them to fight. Depending on the user treats the two Petz, she can try to push them into a buddy or enemy relationship.

The Emergence of Story

Once you create characters that are very reactive, interactive, communicative, expressive and adaptive, exciting things begin to happen. You can suspend your disbelief and become immersed in the illusion of life. But even beyond the feeling that you are interacting with something truly alive, we have found that the interplay between a variety of rich archetypal personalities can give rise to many dramatic situations. Small “stories” seem to emerge as these complex synthetic characters act out their innate personalities.

For example, a tired old bulldog won't be able to go to sleep when a young playful kitten is bouncing around him. Because the dog has a nurturing relationship towards the kitten, he will bring the kitten some food, which then allows him to lie down for a nap. However the kitten, by its own nature and personality, has a short-attention span and quickly returns to jumping around the bulldog. The bulldog expresses his frustration at being tired, but because the kitten is gesturing for attention, he responds by bringing the kitten a toy. Note that the bulldog was never explicitly programmed to “distract” the kitten so he could sleep – but if you look at these “naturally” unfolding events as a whole, it comes together as a vignette about a dog trying to distract a kitten so he can sleep.

Catz are always on the lookout for mice. If the user takes out the cheese and puts it in front of the mouse hole, the mouse may sneak out for a nibble. The cat will immediately begin stalking it for an attack. If the user then brings out a dog, a three-way chase ensues. The situation usually ends with the dog chasing the cat away and the mouse getting a free meal. This has the look and feel of a live Tom and Jerry cartoon.

Even a loving relationship between two buddies can be upset when the user pays far more attention to one pet than the other. The jealous pet will become angry and upset and may even act aggressively to the user or the other pet. However, given time the two Petz may eventually make up and return to their friendly relationship. Over the long term this can appear to the user as a “stormy” relationship.

On the PF. Magic website6 there is a bulletin board where users have the chance to post stories and photos about their Petz. We’ve had a tremendous response. We’ve received thousands of stories from people describing their relationships with their individual pets. Some kids we know get up especially early every day to feed them. Most of the stories describe feelings and complex behavior that seem to emerge from playing with the Petz. Here’s an example:

Our family just adopted our new puppy (named Spike) a few days ago. He is a bull dogz. We let Spike out all the time, and he is already part of our family along with our (non-computer dog, a doberman), who is being neglected and becomming very spiteful, because Spike is a new puppy and requires more of our time. We have had to be careful, about letting them spend to much time together, for Spike is still a puppy and wants to play and our doberman is extremely jealous and continues to pick fights. We are concerned that our doberman will hurt Spike. When it is feeding time (almost every time we walk into the room, if it was up to Spike) we have to put down a bowl of food for each dog. Luckly they can not reach each others bowls. We are afraid to leave Spike out of his play pen for to long of a period of time, without constant monitoring, for both their tempers seem to be short. We do believe Spike and our doberman will become long lasting friends, as they grow-up together, because they have the same personalities, and are both good guard dogs, and very good pets..... If anyone out their has a bull dogz, Spike would love to hear from you, as our cant read, please send a photo and dog biscuit to sweetpea@inreach.com

These kinds of results are promising for the future of interactive story. It makes sense that this is happening because the core foundation of any story is character. Character is even more important than plot, because without characters, no plot can be acted out. Only by first focusing on making lifelike believable characters will you have the building blocks for creating interactive stories. (Note one should not call this interactive storytelling, because the user is an active participant in creating the story.) A new book by Janet Murray called Hamlet on the Holodeck: The Future of Narrative in Cyberspace goes into great detail about these ideas. (7)

Conclusion

Computer games have yet to incorporate very lifelike, believable characters. Once this finally happens, interactive entertainment will no longer be just the domain of today’s “gamers”. Little girls and grandparents and 50-year-old businessmen will become interested in computer games when they can interact with engaging characters. In the long term, our industry will expand to rival movies and television as the primary source of entertainment for the majority of people.

With Petz we’ve already seen signs that this is possible – they are widely popular among girls and boys, kids and adults. As you know this is almost unheard of in entertainment software.  People want virtual characters that they can form emotional relationships with. Not only do they want it, but they expect it. The virtual pet craze is just the beginning.

We've found that developing lifelike, believable characters requires people with interdisciplinary skills from both technology and the arts. Until recently, building lifelike artificially intelligent characters has been treated as a computer science problem. Character development, in any medium, is fundamentally an artistic, aesthetic decision-making process. Lifelike computer characters are ultimately perceived through the filter of human interpretation. No computer character can be perceived as truly alive and intelligent unless its builders always keep in mind what the final user will perceive.

Acknowledgements

We thank Rob Fulop, John Scull, Alan Harrington, Jonathan Shambroom, Peter Kemmer, Ted Barnett, Jeremy Cantor, and the rest of the PF. Magic Petz team.

References

1) The Oz Project, Carnegie Mellon University.  URL: http://www.cs.cmu.edu/afs/cs.cmu.edu/project/oz/web/oz.html

2) Improv, NYU Media Research Laboratory.  URL: http://www.mrl.nyu.edu/improv

3) Synthetic Characters Group, MIT Media Laboratory. URL: http://lcs.www.media.mit.edu/people/bruce

4) Virtual Theater Project, Stanford University. URL: http://www-ksl.stanford.edu/projects/CAIT/

5) Andrew Stern. Virtual Characters, Virtual Pets, Artificial Life, Artificial Intelligence, Interactive Story.  URL: http://www.netcom.com/~apstern

6) PF. Magic. Dogz and Catz , Your Virtual Petz.  URL: http://www.petz.com

7) Janet Murray.  Hamlet on the Holodeck: The Future of Narrative in Cyberspace.  New York: The Free Press, 1997.