Beta Update 9/23/2011

Hey everyone,

First, I want to thank you all for joining us on Tuesday and for sticking it out through the first hour or so of rough patches. It was a great test for us and we discovered a lot about our datacenter and technology. We couldn't have done that without you and I just wanted to let you all know that we really appreciate your perseverance.

I wanted to take a moment and let you all know what's going on back at Red 5 HQ and what's happening with the beta testing. Let's get started...

The Big Three
There were three major issues that we feel we have to solve before additional testing can be done. I want to lay out what we've identified and steps we're taking to resolve those issues. The three major issues are: server load balancing, nil error on login, and texture streaming.

The Server Apocalypse
Before I dive into discussing our servers and load balancing, I want to clarify that when most people say "server" they are referring to a single physical piece of hardware. In the case of most games though, a "server" is actually a group of servers acting as one, or what is commonly referred to as a "cluster" or "shard". For Firefall, we have a large cluster of servers that players are scaled across using load balancing. The idea of load balancing is that the servers all report up to a front-end server providing information, called a "heartbeat", that says how much more load they can take. The front-end server then assigns players based on that information to make sure that one server never gets overloaded.

In the case of Tuesday's test, the heartbeat information was incorrectly saying that the servers were under no load and to keep piling on users. This caused a single server to get overloaded which caused a chain reaction of events that caused a myriad of problems from people not being able to join a matchmaker, to world servers crashing, to inability to log in, etc. It took a while to figure out all the symptoms were caused by this bad heartbeat and then a little more time to work around the issue by changing the way the load balancer selects servers.

In the end, we were able to get most people playing and things balancing relatively well. At this point, we feel we can load balance properly again while we continue to work on correcting the heartbeat problem so that the system can do everything it was meant to do.

The Secrets of Nil
The Nil error that showed up in people's consoles while trying to log in caused all kinds of headaches and nearly made afex pull out all his hair (which probably would have caused much rejoicing around the office, but oh well). He spent all night on the issue, but was only able to narrow it down to packets being malformed (containing incorrect data) when they arrived at the server.

The next two days were spent working with all the networking and server guys trying to track down the problem. The good news is they were able to discover the problem and low and behold, it was actually related to the server load balancing issue. Clients were being assigned a particular server to connect to and even though that server was overloaded and removed from the rotation, the clients were still trying to connect to that server and their packets were being corrupted.

The only way to fix it on Tuesday was to completely disconnect all connections (which is why removing routers, rebooting, etc helped) and even then you had a 50% chance that you would still connect to the bad server. In the end, they were able to reproduce the issue and are putting fixes into the client to make sure that it doesn't happen again.

Attack of the VT
The final major issue that we want to solve is the streaming of textures. First, texture streaming is very important to the game overall. When you have a massive open world, a free to play game, and a downloadable distribution; you want to make sure you have the initial download size as small as possible and make sure players only get the files they need rather than wasting time downloading up front. You'd be amazed to see the drop off data of people for every extra minute they have to wait to start playing a game.

In any case, the technology is important to Firefall. The implementation though needs a little work. There were multiple issues that affected people differently. For instance, there is no limiter on download speed for textures, so it attempts to download all levels of the textures as soon as it needs them and on slower connections, that can bottleneck everything and cause severe lag. We are working on adding both a limiter as well as only downloading the texture quality that you need rather than all levels of quality. We'll need to test both these out to make sure everything still functions the way we want. We are also looking into adding an option to the patcher to download all textures up front for those people that want them.

Another issue was that the textures were using up too much space, more space than the original files even. One of these problems was identified as the file being written incorrectly and a file that should actually be mere kilobytes was taking up multiple gigabytes on disk. That is being corrected as well. The final issue that is important for textures is the write location. Many of you mentioned that you did not like the fact that textures were being written to the C: drive. We are working on making it easier for you to change the location of where these files are written. Unfortunately Windows security settings prevent us from being able to write to the install directory in most cases. Microsoft actually recommends you write to the appdata folder for most things.

In any case, you can change this via your firefall.ini file, but we want to make it easier for users either during the install or after install to change the location of these files. This is something that is being debated right now.

The Next Test Cometh
You all provided a great deal of feedback. That is awesome. We are still sorting through a lot of it and discussing plans for some of the more egregious problems. Keep that coming. In the meantime, we'd like to test again with the same group of people and with pretty much the same client (only focusing on the fixes I mention above). Once we've tested the fixes and have a patch ready, we'll schedule out the next test.

The current plan is to test sometime next week. As always, as soon as I have a firm date, I'll let you know.

Thank you again for all of your testing, feedback, and patience. Feel free to let me know if you have any questions or concerns.