Offworld Trading Company 's Early Access campaign

The most common problem in the games industry is waste – wasted time, wasted effort, and wasted money on design ideas that aren't actually fun in practice.

The most common problem in the games industry is waste – wasted time, wasted effort, and wasted money on design ideas that aren't actually fun in practice. Often, this discovery is not made until shortly before shipping when the game is finally played outside of the development team. Basic assumptions about how the game should be played might be wrong, and a community more dedicated to winning can easily find holes in the balance.

No one knows a game both better and worse than the development team, which understands why every decision was made but is also blind to how the game appears to new players.

At Mohawk, we believe that games need outside feedback as soon as possible. I saw this first-hand with Civilization 3 and 4; the former had no external feedback before shipping and thus had numerous gameplay and balance issues that would have been easy to fix if we had simply known about them. In contrast, we recruited a private external testing group from the community to play Civilization 4 over 18 months before we shipped. The logistics of managing this group - with NDAs, physical copy protection, and bi-weekly patches - were a nightmare, but much of what went right with the game can be traced to feedback from this group, which kept us on the right track.

Thus, as soon as I heard that Valve was starting an Early Access program, I knew we wanted to take part with Offworld Trading Company . Getting good feedback from players before release is a logistical challenge, especially for a game with a major multiplayer component, and Early Access would solve that problem for us, a small indie team making a very unusual RTS without combat. We were worried about the potential marketing impact of Early Access on our final release launch, but we still went for it, assuming that the increase in quality from early feedback would outweigh the cost.

WHAT WENT RIGHT
1) Learning About the Game

Feedback is important because it is the best way to learn about a game - finding out how people actually play instead of how the team imagines they are going to. Offworld was on Early Access for 14 months - approximately half of the project - and we learned many things that we would have never discovered internally. A great example was player dissatisfaction with scanning the map before founding an HQ; this feedback led directly to the development of the Reveal Map option that completely changes how the game begin.

"There is no better argument for Early Access than learning about a problem while there is still plenty of time to fix it."

We discovered this issue during the first competitive tournament as the scanning system quickly became a point of contention. The players argued that if a map had a founding location which was superior to all others, the game would be won simply by whoever discovered that founding location first. These players were concerned primarily with a sense of fairness, which was a reasonable concern for the hardcore community because founding location is so important for high-level play in Offworld .

The solution was to start with the map fully revealed and then let players choose where to found, with a debt auction determining who gets to found first. (A counter starts at $200K debt and then goes down in real-time so that players who found earlier start the game with more debt, essentially “buying” their founding location on credit.) This option worked perfectly for our most competitive players and quickly became the de facto standard for online play.

However, the important point is that we made this change a full year before we shipped Offworld , so we had plenty of time to test and balance the mode, write AI for it, and decide how to introduce it to the player. If we did not have Early Access - even if we only had a small private beta - we would not have discovered this important issue until it was too late. There is no better argument for Early Access than learning about a problem while there is still plenty of time to fix it.

2) Live Experimentation

One crucial aspect to doing Early Access right is figuring out how to update the game while also keeping it playable. A good example of how this can go wrong is the Corpse and Hound update from Darkest Dungeon (see Tyler Sigman’s 2016 GDC postmortem) which put the developers at odds with a vocal portion of their community. This type of conflict is paradoxical – the point of Early Access is to be able to change the game for a live audience and yet players can punish developers for doing exactly that.

"Being able to experiment rapidly and without fear was a major factor in taking advantage of Early Access to improve the game’s design."

We were very careful about how we rolled out changes while also maintaining the position that the point of Early Access was live experimentation, so we would be fearless in that regard. We took a number of steps to meet these two conflicting goals. First, we released major updates slowly so that casual players would not experience random bugs during normal play. Then, to get feedback on our most recent changes, we created a Steam branch entitled “next_version” which we updated multiple times per week. (This branch was password protected, but we shared the password publicly so that it was essentially an ongoing opt-in patch for our hardcore community.) We felt free to make any changes we wanted to on this branch; if players were upset by a change, they could always just switch back to the main version. Our core community knew that we wanted to hear feedback about this version, so they were excited to jump onto the branch, see what was new, and let us know how they felt about it. Most importantly, they were never blindsided by a change because the next_version branch was constantly updated.

However, when we made potentially controversial changes, we would attach them to game options that the player could disable. For example, our stock system underwent many significant iterations, with some of the changes being more popular than others. In one patch, we added two major features - Destroy Buyout (a player’s buildings are destroyed on a buyout) and Majority Buyout (a player is eliminated when more than half of his or her stock is owned by rivals) - but both were options that the could be turned on or off. Thus, players who hated the changes could play without them while we keep experimenting. To ensure that we would learn enough about these new features, we hosted a community tournament after releasing the patch which specified that both options must be turned on.

Knowing that an upcoming tournament would use these rules encouraged our players to practice with them in preparation, which produced meaningful feedback for us. (In this case, Majority Buyout became a standard rule while Destroy Buyout did not, being replaced in the long-term by the Subsidiary system.) Being able to experiment rapidly and without fear was a major factor in taking advantage of Early Access to improve the game’s design.

3) Building the Community

A vibrant community is important for the long-term health of a game, especially one with a strong multiplayer component. One of the great advantages of Early Access was that we were able to build that community before launch. Although players still find each other on forums, we found that the best place for a community to form is on Twitch. We discovered our best players early by seeing them play on streams, usually ones involving multiplayer games.

"The quality of a game is determined by how many times it can go through the design, code, play, and listen feedback loop. With Civ 4, the best we could do was bi-weekly; with Offworld, that loop could be daily."

We encouraged that growth by hosting Twitch tournaments, meaning that after we organized the brackets, players were required to stream their games. (Because some players were not capable of streaming, other players jumped in to stream the games as observers.) Players and viewers would jump from match to match as the tournaments progressed, forming bonds with each other in the process.

Long-term, our community became an important part of our development process. For example, when working on the AI, I would ask players to do their best on the Daily Challenge (a random map based on a new seed each day) and post their videos or replays online. I would watch to find the biggest holes in the AI’s performance, write some code to fix things, push a build to next_version on Steam, and then ask players to try again.

The impact of this rapid iteration cannot be overstated. The quality of a game is determined by how many times that game can go through the design, code, play, and listen feedback loop. With Civ 4 , the best we could do was bi-weekly; with Offworld , that loop could be daily.

Finally, building a strong community during Early Access paid off handsomely at launch because we had already primed an active group of players. On their own, they organized a 24-hour marathon of veteran players streaming the game for newcomers interested in the game. Two of our best players - Zultar and Cubit - wrote comprehensive strategy guides that we included as free DLC. Many YouTube videos were already online for players who wanted to see the game; according to Steam Spy, these videos averaged over 200K views per day during the first week. Our final release launch sales were just as strong as our Early Access launch sales, and much of the reason was having an active community already in place.

WHAT WENT WRONG
1) Constantly Shipping

Over the 14 months of Early Access, we shipped ten major updates to the game along with a number of hotfixes, which absolutely took its toll on the development team. Each update had to go through a round of QA, with bugs being assigned to developers who had to interrupt their normal development flow to ensure the update was polished and ready. Some of these bugs were critical, but others were of subjective importance. The QA team was trying their best to be thorough, but during active development, not every bug needs to be fixed, especially for systems that are currently just placeholders. I gave each team member the right to make a judgment call on which bugs to ignore, but the process itself absolutely took time away from more important, long-term tasks.

Also, being on Early Access for half of the game’s development cycle meant that we couldn't include half-baked features with just debug text and programmer art. This type of prototyping is an important way to make progress while a feature is experimental so that polish would be a waste of resources. Sometimes, we would lock away features that we knew were not ready for a general audience by enabling them only in special developer builds, which helped mitigate the problem. Regardless of the Early Access label, the general Steam audience is simply not ready to see just how ugly games can be during development.

2) Steam Reviews

Although Steam tags user reviews written before release with a special “Early Access Review” designation, these reviews still count against the game’s positive review percentage. We had generally positive reviews, but - as our current percentage is just two point shy of the 80% threshold for the Very Positive status - it is hard not to imagine that we would be in a higher category of we weren't saddled with user reviews written 14 months before release. (Our Executive Producer at Stardock, Derek Paxton, took the time to respond to every old negative Steam user review that had a specific complaint addressed in the release version, and we did see a number change their review.)

On a personal level, negative Steam reviews took a not insignificant toll on my own personal morale, and the same is probably true for the rest of the team. It's hard to read over and over again that the game doesn't have enough content, has crummy voicework, is missing a tutorial, and so on – as the team is working to fix those issues. Even with an imaginary, unlimited budget, one has a finite amount of energy to invest in a project, and premature yet permanent ratings can drain that energy surprisingly quickly.

3) Press Apathy

One common argument against Early Access is that “you only get one launch” – meaning that a game’s Early Access launch is, in truth, it’s only launch. We certainly experienced a surge of interest in Offworld when it first launched in early 2015; our game was the new shiny object, so we were able to organize a media blast by revealing the first screenshots a couple weeks before the Early Access release, resulting in exclusive stories and interviews on Gamespot, Polygon, IGN, VentureBeat, and more. Shortly after release, videos appeared from popular YouTube personalities like Sips, Quill, Arumba, and Northernlion while huge audience watched streams from Trump and Day[9].

"We proved that a game can have a second launch, although we certainly felt like we had significant headwinds from a lack of mainstream press interest"

This wave of media interest led to very strong sales during our first few weeks on Steam even though we launched at $40, making Offworld one of the highest-priced games on Early Access. (At launch, only Galactic Civilizations III - also published by Stardock - was more expensive.)

14 months later, however, we had a much harder time getting press attention for the game’s real launch; most of the websites who wrote about the Early Access launch told us explicitly that since we had been on Steam for so long, they didn’t find us newsworthy. We could expect reviews but little else.

Fortunately, our reviews were strong (82 on Metacritic, including a glowing reviewfrom Rock, Paper, Shotgun), the game had been on over 200K wishlists, and forums activity around the game was high, so we ended up selling nearly the exact same number of copies during our second launch as compared with our first (two-week sales of 23,607 vs. 23,457, respectively). Thus, we proved that a game can have a second launch although we certainly felt like we had significant headwinds from a lack of mainstream press interest.

We are not entirely sure how we were able to sell so well the second time without a strong media push. I’d like to think that the tradeoff we knowingly made by going on Early Access - sacrificing a traditional media buildup for the benefit of early feedback to make a higher-quality game - paid off in the long run although that is, of course, impossible to prove. Indeed, it’s still possible that we may have sold more copies without Early Access if we had paired our media announcement blast with our polished release version, but we’ll never know for sure.

WHAT WE WOULD DO DIFFERENTLY

Without question, we would be willing to put our next game up on Early Access as well. However, we will certainly approach the process differently.

First, we would try harder to sell our game independently before going up on Early Access. Many games do this ( Factorio , for example, sold over 100K copies before launching on Steam), and we sold Offworld with a “Founders program” on our own site but without actually showing any screenshots or video of the game. Users were buying the game blind, and predictably, we sold the game to just a handful of dedicated players. We were worried about releasing images while the game was full of prototype art (and inevitably seeing those images stick around forever online), but developing a game publicly before going up on Steam has huge advantages.

"Next time, if we are getting sufficient feedback while selling the game independently, we would delay our Early Access launch as long as possible."

The pre-Steam audience is more mature and more forgiving of the inevitable bumps of game development; they know upfront that they are buying into something speculative. During our Founders program, we uploaded builds without running it through QA at all (although we often ran a test game internally) because we knew this group would tolerate a few hours downtime if we released a bugged version. We had the freedom to develop without worrying about a temporary mistake landing on our permanent record via negative user reviews.

Next time, we would accept the downside of showing prototype art publicly to allow us to stay off of Steam longer; if we are getting sufficient feedback while selling the game independently, we would delay our Early Access launch as long as possible.

When we do launch on Early Access again, however, we would do a few other things differently. We would actually drop QA from the process entirely (at least during the long period after the initial launch and before final release) and instead develop a build process that fits the needs of the development team, which should always be thinking long-term and not worrying about a bug list too early, as well as the Early Access audience, which both wants the game to work yet also to see frequent updates.

One possibility is to run a build automatically every Monday morning, which is then uploaded to Steam without testing. This build would be sent to a branch similar to next_version (although this time, the branch would be public and not password protected), which our more dedicated players would try out right away. If there are any problems during the week, we can manually update that branch. By the following Monday, if next_version was stable, we would promote it to the main branch for general consumption.

Hopefully, this development process would allow the team to work unhindered while also making sure that the players are experiencing a current version of the game. Indeed, one of the major problems of releasing monthly updates is that even after a single week, players might be playing a game already outdated by various new features or balance changes. This makes their feedback -- which is the whole point of Early Access -- no longer relevant. (Note that some Early Access teams argue that major, themed updates produce the best sales, but we are assuming that sales figures should not be a priority during active development.)

WHAT STEAM SHOULD DO DIFFERENTLY

Most of the problems with Early Access result from differing expectations between developers and consumers. Developers want Steam to provide a safe place to build a game while also exposing the game to a large enough audience to get worthwhile feedback. Consumers, on the other hand, want to play a game early and, hopefully, for a low price as well. (However, we heard many times from players that they didn’t even realize they were buying a game on Early Access or what that designation even meant!) We have a few suggestions for improving the role Steam Early Access plays in game development.

1) Allow unlisted pages

The biggest benefit of being on Steam during early development is not the exposure but the infrastructure: build distribution and branching, access to the Steamworks library, free community support features, and online sales through a trusted store. All of these tools were major problems for independent developers ten years ago and are now easily solved via Steam.

" Valve has become increasingly resistant to allowing developers to sell Steam keys without their games actually being for sale on Steam. "

However, Valve takes an all-or-nothing approach to Early Access; launch will put the game on the front page of Steam whether the developers want it there or not. During our Founders program, we sold Steam keys directly through our website, but Valve has become increasingly resistant to allowing developers to sell Steam keys without their games actually being for sale on Steam.

This policy is forcing some independent developers to other options for their first online sales. (Adam Saltsman launched Overland ’s “First Access” on itch.io for this very reason, and they are even limiting the number of keys available for public sale.) Most developers would prefer to stick with Steam (as it has the most mature infrastructure) but are afraid of risking their reputation by launching too early.

The answer is to allow developers to sell games on Steam with unlisted store pages, meaning the page is only available via a direct link and does not show up in any advertisements, ranked lists, discovery queues, curator collections, or any other method for exposing the game to the average Steam consumer. This option would allow developers to start selling their fledgling games slowly while still benefiting from Steam’s infrastructure.

2) Unscored user reviews

User reviews are a staple of online commerce, and Valve was wise to implement them, even with the potential chaos inherent with giving customers the power to judge a game anonymously. However, what exact purpose does a review serve when stating that an Early Access game is not ready? The game’s presence on Early Access is an explicit statement that the game is not ready!

More importantly, the existence of scored user reviews argues that Early Access games should be judged and evaluated the way the normal games are, which is simply not true. If the team is serious about iterative development, Early Access games can and should take wild swings in quality during development; the fear of negative user reviews encourages developer to sit comfortably on local maximas.

Removing scores (meaning thumbs-up or thumbs-down) from user reviews should send a message that Early Access games are not ready for a final evaluation; the goal is not to trick people by removing scores but to sell to less people, the ones who are onboard with experiencing an unfinished product.

3) No sales, No refunds

Implicit in the argument against user reviews is the belief that Early Access would be healthier if the games sold to less players overall but also to ones who are more dedicated. Turning off the developer’s ability to reduce the price of a game to drive sales and turning off the consumer’s ability to test out a game knowing that a refund is possible should both drive down game sales, especially among the more casual audience looking for either a bargain or a de facto demo. A Subnautica developer spoke to their view on driving sales during Early Access:

Subnautica is a game that is still very much in development, and we don't need to bring in a large influx of players right now. When the sale price is lowered by a large margin, it tends to attract a group of people who are less willing and dedicated to giving the game a real chance. You end up with players who just tossed it on the pile of other games they are buying, mainly because it was a great deal, and many of those people either never end up playing it or end up playing it for a short amount of time and posting a negative review because they likely didn't research it.

One potential alternative that would provide some flexibility in driving sales but without bringing in players not ready for Early Access would be to allow two ways to buy the game - a pre-order option and a play-now option. The latter could be slightly more expensive, which should keep away players who are looking for a deal, while the former provides a nice way for consumers who trust a developer to support them.

Ultimately, Early Access development would be healthier with a slower and steadier influx of players, growing the old-fashioned way by word-of-mouth. We want a special type of consumer, one who is excited about seeing behind the curtain, contributing critical feedback, and seeing the game evolve. We have met these types of players over and over again online; they inspire us with their passion and patience while we work hard to build the best game possible for them. These players are priceless, and Early Access would be the best place to find them if Steam is willing to do the work to guide player expectations.

Article updated 6/23/16 - changed "No User Reviews" to "Unscored User Reviews"