MMORPG Anatomy - Gamedev.org wiki

The MMORPG Anatomy series of entries attempts to diminish failure rates in independent game projects by offering a simplistic, yet powerful series of arguments against the creation of massively multiplayer online games of all stripes.

It tries to avoid overly technical or platform-specific explanations, instead focusing on the logistical and management aspects of developing a large scale MMORPG. It assumes you have no less than 5 years game programming experience.

Please note: By massive we mean a game with over 1 000 simultaneous players. Other people use a definition of over 10 000 simultaneous players.

• If you aren't willing to do the things in this article, it may be possible to reduce your scope, target a smaller number of players (200-300) and produce an online RPG that is stable and powerful on one machine without the massive support issues that sprout up from heavy load.

Table of contents [showhide]

1 Why are MMORPGs so complex? 1.1 Development 1.2 Server management 1.3 Userbase Support 1.3.1 Security/Griefing 1.3.2 Responding to criticism 1.3.3 Managing cash 2 What does an MMORPG look like inside? 2.1 Client side 2.1.1 Graphics rendering 2.1.1.1 Security Implications 2.1.2 Data management 2.1.3 Net code 2.2 Server side 2.2.1 Persistence 2.2.2 Security 2.2.3 Cheat prevention 2.2.4 Load balancing 3 How can I reduce MMORPG failure rates? 3.1 Reduce Scope 3.2 Buy vs. Build 3.3 Hiring Right 3.4 Code Quality 4 Why are MMORPGs so popular for indie teams?

edit

Why are MMORPGs so complex?

MMORPGs are complex beasts, and the complexity doesn't stop after development has ceased and you have your interim release out. Online development means that you need to be a master of system administration, security management and game design, all at once, and that's after the constantly-increasing technical costs of game development.

edit

Development

Some points:

• Development is getting more expensive. Now that we have higher-definition hardware with more powerful resources, we need more content. That content requires more developers. Hence, game developers are spending a lot of money to build commercial level games.
• If you're reading this page, you probably don't have much money to spend on this game. We're talking millions as the barrier to entry for a commercial-space MMORPG.
• Take one programmer's average salary, multiply it by the number of programmers you need, and then again by the number of years you're in development. If the total is under two million dollars, you don't have nearly enough to consider writing a commercial MMORPG, or you are bad at planning software costs.
• The cost of games is crazy (http://www./content_page.php?aid=21454). Even EA thinks so. And they have a lot more money than you do.
• The basic sum-up is this: Development costs lots of money.
  • You can reduce development costs by reducing scope. An independent team targeting a commercial MMORPG is stupid and insane, and will almost certainly fail.

edit

Server management

A lot of stuff on a decent MMORPG happens server side. This means you need to have very powerful server machines with lots of bandwidth, particularly if your game has any sort of real-time behaviour.

Note the plural of 'machines'; a lot of multiplayer game developers have the single-server model (QuakeWorld), but MMORPGs require several different machines, often load-balanced using complex algorithms and real-world performance testing. The networking code in a large scale MMO is highly tuned, tested extensively, and designed to have reliability and performance that is rare to find even in large enterprise application development. Websites can get away with a bit of lag from time to time -- real-time games cannot.

Obviously, you'll need to manage those machines, and pay for the bandwidth. If you're just starting out with a small game, you might be able to keep 50-60 users happy with a single dedicated server and pipe from a place like Dreamhost for a few hundred bucks a year. If you have plans to expand, you will move to several boxes, very quickly, and your costs will reflect that.

Plan to expand, but also plan to spend a lot of dough on hardware and hardware management, or to hire someone full-time to take care of it.

edit

Userbase Support

MMORPGs obviously need users. You need to keep these users happy, or they'll go to some other game. This is bad, particularly if you're expecting those users to give you money or ad impressions to keep your server alive.

edit

Security/Griefing

It would take a whole other article to discuss Online Game Security, so this is just going to be a quick draft of things you need to watch out for.

• Client trust attacks: The player "lies" on behalf of the client, and your server trusts the client that it is correct and not cheating. This is where you see many of the issues in older MMORPGs ("speed hacks", etc). Proper game architecture must avoid this case by checking data received from the client whenever possible. Never trust the client; it is in the hands of your enemy.
• Hidden data leaks: The player sees data that is obscured by the client, but is sent to the client by the server. This is where you can see tricks such as "wall hacks", character detectors and other knowledge that a player is not normally supposed to be aware of. Proper game architecture must avoid this case by sending the proper data whenever possible.
• Network layer attacks: The player exploits certain properties of your game's lag prediction or other network compensation features to gain an unfair advantage in the game, perhaps by artificially creating lag or spurious disconnections.
• Macroing: The player takes advantage of third-party software to automate game play, so they can allow their character to complete time consuming tasks in-game while not using the computer (or not actively playing the game). While not a hard and fast rule, allowing macroing does tend to offend your user base. It can be hard to detect as developers of macro products get better at concealing their activities.

Then we have discussions of griefing, which can range from the simple (player murder) to the immensely complex (groups of players exploiting your game features to annoy and harass others in a fashion you never intended). You must respond to these concerns quickly, or your game will become unplayable and your players, expecting administrative intervention, will go elsewhere.

edit

Responding to criticism

Players will have criticism of your game. This criticism may involve:

• Balance/gameplay issues -- "This class is less powerful than that class".
• Service availability issues -- "WTF LAG WTF"
• Feature requests -- Ranging from the simple to insane.
• Other players -- See griefing.

Obviously, part of producing a successful service on the Internet, whether an online store or MMORPG, involves responding to user feedback and handling crises. How should you do that?

edit

Managing cash

If your game costs money, you will have to deal with that money.

There's basically two ways you can handle it:

• External payment management -- You give up a lot of control over the process but don't have to actually bother with it, and there can be many kinds of payment methods supported effectively "for free".
• Internal payment management -- You are going to be spending a lot of time adding new payment options, dealing with crises (such as credit card rollbacks), and exposing yourself to the possibility of legal action to regain "lost" fees. On the plus side, you have maximum control.

My personal recomendation is to go with an external payment manager; you lose control and they often charge aggressive fees but this is the best method. Consider payment managers such as Kagi (http://www./) and PayPal (http://www.paypal.com).

edit

What does an MMORPG look like inside?

edit

Client side

The client side is the easier of the two. This is like saying building a Sherman tank out of wood is easier than building a microprocessor out of sand.

edit

Graphics rendering

This is probably one of the best defined fields of MMORPG development.

Basically, you're going to want to render your 3D graphics at a decent clip and still make the entire thing relatively attractive. While this is one of the single largest problems in game development and consumes the lives of academics for years, compared to the other problems of developing an MMORPG this is remarkably minimal.

Also, consider that to be truly "massive" you'll want lots of users. Lots of users don't have super-powerful gaming boxes. You need to scale whatever graphics you put into the game, obviously.

edit

Security Implications

Yes, there's even security issues with tossing polygons at your GPU; Counter-Strike style "wallhacks" and other exploits work by substituting the video driver with one that doesn't do proper depth clipping. There's also several popular OpenGL and Direct3D debuggers which can capture all rendered geometry and dump it to file; obviously just because the player can't see it right now doesn't mean the polygons will never get seen if you render them behind them.

This is a general multiplayer game concern, though, but is still something to think about.

edit

Data management

The client side has to manage data.

Obviously, character data should be stored on the server, outside of the player's reach. This prevents the player from modifying their character data sheets and obtaining items and powers that they shouldn't have.

What are you storing on the client? Generally, it's stuff like

• Game content -- Cached data, and whatnot. It would be way too slow (and expensive) to download all the game content off the server when the player encounters it.
• Preferences -- Server addresses, client login data, video preferences, etc.
  • You should also consider safeguarding this data -- many exploits have been used in other games to gain the stored client login data of users and drain their accounts with this information.
• Game Executable -- The program and all the scripts associated with your game.
  • You should ensure that these are tamper proof: nothing would be worse than having players tinker with your freely available scripts and developing private servers to subvert your subscription fees.

This also brings up the sub-topic of patching, which is an important concern. When you patch the game, you'll want to enforce it on all players so that their client-side data syncs up properly; this means distributing a new set of hashes as well so the game "knows" when someone has substituted false data.

There have also been some cheating attacks on games previously by modifying the patch as it comes down off the wire.

edit

Net code

Obviously, network code is very important for the client. Changes you make here will directly affect the performance of your game, and obviously the performance of your server.

Some key points to keep in mind:

• Smaller is better. Keeping the amount of data you send to and from the server to a minimum will help save on bandwidth costs, reduce lag and clean up your code.
• Be Paranoid. Remember that the client is always in the hands of your worst enemy.
• Consider the Edges. A good network programmer understands the properties of the networks he is operating on. Be aware of the consequences of congestion, hardware failure, low bandwidth environments and other concerns, since the network connection is only way the client can actually play the game.

Other concerns about net-code are game specific, but this is a good start point to begin thinking about the places your client's net-code can go wrong.

edit

Server side

In an MMORPG, all of the gameplay and events occur server side. Computing gameplay consequences client-side is an invitation for cheats.

Since you're handling a pretty beefy number of players and entities (with complex behaviour) you need to be sure that you're handling them quickly, in order to reduce lag and ensure a fun gameplay experience. This is easy enough when you have a small game, but as you move towards "massive"-level games, you will find that the obvious, dumb and slow approaches no longer work.

When developing server code, in addition to the following points, you should be designing your code to:

• Be multithreaded: Most high-end servers nowadays are multi-processor or multi-core. Having a handle on concurrency also helps when you have to load-balance and split across servers (see below).
• Be lightweight: Operations that take up several megabytes of memory can quickly sap a large server under load. In addition, troublemakers may take this opportunity to perform those operations repeatedly in order to deliberately bring down the server.
• Be paranoid: Check user data constantly -- assume that the client is always in the hands of your enemy. Also check other data constantly -- be aware of corruption, security concerns and hardware failure.

edit

Persistence

You have to save data, obviously. Where do you save that data? In most implementations, it's to a database.

But there are several concerns relating to a database:

• Concurrency issues with multiple servers (especially with multiple database servers)
• Additional processing overhead
• Backup concerns
• Reliability concerns
  • Data integrity
  • Server stability
  • Atomicity of operations

If you don't have experience with databases, it's highly recommended you learn about many of the solutions out there and begin to develop a plan for your game's data persistence. Flat files are completely unacceptable for an MMORPG, and even your average off-the-shelf database will buckle under a truly "massive" load.

edit

Security

You must protect your server and server code itself against security issues. In the corporate world, this is the work of a full-time team of system administrators for a reasonably sized data farm.

Some of the concerns you should be aware of:

• OS-level vulnerabilities: Operating system misconfigurations or application misconfigurations can lead to security consequences you are not aware of. Keep operating system and application software patched and monitor it closely.
• Game Server-level vulnerabilities: Nobody is perfect. There is most likely at least one security flaw in your game's server code. Exploiting it can lead to severe consequences for other players, and for you. This isn't even counting client-side cheats, which are described above.

There are also edge conditions (physical attacks on the machines, management console attacks, ISP attacks) that must be understood if you are to provide a robust service on the Internet.

edit

Cheat prevention

Really an offshoot of security, you need to be aware that the client can and will feed you bad data if they get the opportunity.

Doing things like confirming input (can the player really walk to this destination, or is there a rock in the way? Did they "warp" too far?), discarding malformed input, and quickly fixing gameplay bugs can help eliminate a lot of the cheating.

However, also consider that your gameplay mechanics themselves can give rise to cheating and/or abusive behaviour (such as griefing). Ultima Online, for instance, was plagued by a series of robberies by players who determined they could painstakingly stack items to build a stairwell and lay siege to the homes of other players.

edit

Load balancing

If you run your game on a single server, you will begin to notice problems once your load starts to creep up. This is generally due to a bottleneck in your hardware, bandwidth, or something else.

In the olden days, we used to deal with this by buying specialized hardware, optimizing the server code and spending hours upon hours poring over hand-tuned router rules. Nowadays, that isn't practical. The ballooning number of configurations and the success of Google have shown that commodity off-the-shelf hardware is probably your best bet (although, don't forget to optimize your server code anyway).

So, that means you'll be adding a new server, pretty close to your existing one. That means that you now have two servers, which must balance the load of users without sacrificing their gameplay experience.

In the past, attempts at load balancing generally split the world into separate "realms" of different users and rules. There are some newer designs that reduce this amount of splitting, but they are still load-balanced across multiple servers.

In the end, you're going to end up with multiple servers, and probably with multiple database servers. This is an entirely different class of scalability concern, and you're going to hit this as soon as your game becomes relatively popular. It's best to plan for this from the beginning.

edit

How can I reduce MMORPG failure rates?

Like any complex piece of software, the risk of failure on an MMORPG project can be mitigated by following good business, software design and development practices.

For more information on why games fail, consult the Game Engineering page.

edit

Reduce Scope

Cut your design down to fit the time you have to develop the software, even if it involves cutting one-off features. Set a design in stone, and don't move from it.

edit

Buy vs. Build

With so many complex components moving about inside your project, it will take many years to develop all of the components of the game from scratch. Many developers are opting instead for middleware for various components of the game engine.

Some of the benefits of a middleware solution:

• Faster development speed.
• No re-inventing the wheel; many middleware solutions come with source code that can be altered for your game's purpose.
• Technical support from the middleware vendor.
• Depending on the middleware, you can get production and publishing assistance with your product.

Some of the downsides:

• Licensing can be expensive, particularly with royalties and large sales.
• Middleware vendors can disappear.
• Interfacing with the middleware can be complex; ideally you want to develop from the start around the middleware instead of attempting to shoehorn it into existing code.
• Documentation for middleware can be shoddy. Make sure you know what you're getting into before spending the money.

Many of these arguments also apply to game industry contractors; obviously, contractors are of varying quality and discussing the HR issues related to them would take up an entire other page.

edit

Hiring Right

• Don't bother hiring anyone from the Internet.
  • Especially for free. There is virtually nothing keeping those people from wandering off, or stealing your code and assets as they please.
• You want lots of experience, ideally related to the realm.
  • Don't hire friends.
• Before going shopping for team members, make sure to have a feasible demonstration, lots of content, and a clear indication that your business is healthy.
• Attend industry trade shows to help attract the better developers.
• Don't just hire more people. Increasing the size of a team can negatively affect development speed.
  • See the Game Engineering page for more details on software engineering theories and how they can be put to use in independent game development.

edit

Code Quality

With so much code, an MMORPG is sure to be prone to failure.

• Use good software engineering practices: unit testing, documentation, everything.
• Make sure to perform constant code reviews. Well-done code reviews turn up strange bugs and edge cases that the original developer(s) of the code cannot always see.
• Ensure your code is as clean as possible; refactoring mercilessly (http://www./rules/refactor.html) helps but is not a Silver Bullet for making your code adequate.
• Buy the best possible equipment and tools for your developers.

edit

Why are MMORPGs so popular for indie teams?

MMO's are extremely popular amongst indie devs because a lot of indie developers are young, inexperienced or both. This is not a bad thing to be; we were all young once and we all started with little experience. The problem with the MMORPG genre is that a lot of what is going on is occuring server-side. The gamer doesn't see it. So when it's done well, it looks like the game is actually quite simple to make, which can mislead the inexperienced. The truth, however, is that they are one of the most complex genres available today. They encompass multiple facets of the programming spectrum -- most of which are specialty fields in their own right.

The reason experienced indie teams (NOTE: teams) take on MMOs is generally because they like the challenge and having an open-ended project means they always have something to tinker with. However, it's very rare you'll find an indie team trying to make something that competes on a commercial level.

There are reasons for this:

• It's not really feasible on their budget,
• They know they simply can't compete with the seemingly endless resources of major commercial development studios,
• They don't have the time available to develop a commercial level game in any reasonable amount of detail,
• They have experience with managing complex systems and foresee many pitfalls which may be caused by their current inexperience.