PR Master Server Update
Posted: 2016-02-07 04:40
Hi everyone. Time for an update on the current situation with the master server.
Firstly, it's back up. You can all go play now! Yay!
But for those still reading, I want to give you all an overview of what happened over the last week, and what's happening in the future, so this doesn't happen again (or so it doesn't happen as bad again). To start with though, I want to apologise to everyone for the downtime. It sucked, and I know how much everyone was wanting to play, but couldn't.
What is this Master Server?
The master server is what controls your online player logins, and the ability to see servers in the ingame server list. This service used to be provided by GameSpy, and many games used this. However, in 2014, GameSpy was shut down, and Battlefield 2 became abandonware. To keep PR alive, we reverse engineered the original GameSpy master server and launched a replacement service. Also Forgotten Hope 2 was invited to use our master server as well, in the spirit of community relations.
What happened to it?
Last Saturday, the master server was attacked by a DDOS attack. This caused our server provider to null route the IP address running the master server, effectively shutting it down. When the master server is down, no one can play PR online. Also, since FH2 is using the master server, they were also affected by this downtime. Obviously, quite a bad situation to be in. This wasn't a new thing to us though, it's happened several times before, where we've had outages for a few hours here and there.
Why did it take a week to come back up?
Shortly after the DDOS took down the master server, I contacted the server provider to request more information, and to release the null route, like I normally do. Unfortunately, this occurred on a weekend, and the Tier 1 support I contacted could not provide any assistance. They referred me to Tier 2 support, who unfortunately only work Monday-Friday. Not the sort of thing you want to hear. I did get some good news on Sunday though, the Tier 1 support person finally realised it was an automatic null route, and suggested that it would return within 24 hours from the original attack. Unfortunately this did not occur, so it was back to waiting until Monday.
When the server did come back up, it was just attacked again, and the automatic null route was reinstated. That was something new that we haven't had before. The attackers usually gave up after a few hours. On top of this, our website was attacked and taken down as well. So this attacker was quite persistent.
To make matters worse, I personally had to leave to go on a business trip Sunday-Thursday, and was unable to do much else apart from submit support tickets from my phone to try and get the master server online again. Unfortunately, I did not have access to do the same for the website, and the person who does failed to reply to our messages. We did eventually manage to track down a retired developer who did still have access, and we got the website back up.
The master server however was continuously attacked over and over, something like 5 or 6 times if I counted correctly, and I gave up trying to get it back up until I returned home to deal with the situation properly.
So what has been done about it?
Yesterday, we have moved the master server to a different provider, one that implements much more effective DDOS mitigation techniques. We will see how everything goes over the next few weeks, and if the attacks continue and are mitigated, or if they prove ineffective. If we continue to have issues, then we still have some more things that we can do. Changing server providers was just the easiest and quickest solution. We had hoped to get it all up and running yesterday, but we ran into some slight issues with the migration, so we had to delay until today.
What about the website and that attack?
While the master server is the more critical part of the picture, the website is just as vulnerable to this type of attack. Over the next few weeks, we will be working on migrating the website to a new server which will also be able to mitigate these attacks, as well as moving all our other services that run behind the scenes. This new server is also cheaper and more powerful. Which brings me to my next point.
What's up with donations?
As I mentioned last week, donations at the moment only pay for our website, and some other behind the scenes things like our repositories. There are two servers that cost us $319 USD / month, and the donations pay for these. On top of this, I have my own server, which provides services such as our build server, test server, file hosting, project management tools, mumble, PRSPY, various utility scripts, and up until yesterday, the master server. This costs me $99 USD / month, which I pay for out of my own pocket, and have done so for the last 5 years or so. There is also a CDN which I also pay for which hosts some critical things, but that's like $2 / month, so it's not really worth mentioning.
Anyway, for those who know how much these sorts of things cost, it's clear our costs are way over the top, over $400 / month, and for the last 3 years or so we've been trying to consolidate servers and reduce our costs. Unfortunately, our web admins have been pretty inactive, and haven't been able to get anything done about it.
So instead of waiting for the web admins, we are now actively working (over the next few weeks) to just get this done already. It's been pushed back and back, and with all these attacks it's finally pushed us over the edge where we can't deal with it anymore.
Unfortunately, issues like inactivity are a common part of working on a volunteer mod development team, where you don't get paid and you work in your free time. We just have to deal with it and try to overcome any issues that may arise.
So, we are purchasing new infrastructure to handle all our use cases, consolidate all our over-priced servers, and avoid the need for developers to pay out of their own pockets.
So how much will these new servers cost?
Our new planned server infrastructure will cost us less than $200 / month, for everything, which is quite a large difference. Over the next month or two, we will also be sorting out donations to make sure it covers everything properly, unlike before (it takes time to deal with this sort of thing properly, it's not something we can just flick a switch on). Our current donation goal of $250 / month will likely stay as is, so we can cover months with reduced income.
For those wanting to donate now, please hold onto your money for the time being. We will let you know when donations are going to the things you want to pay for.
Another bonus of the new infrastructure will be multiple people having appropriate levels of access, so we don't run into a situation like last week where we can't do anything because certain people aren't responding.
Who did the attack?
Dunno, clearly someone who obviously doesn't appreciate the work we do here, provided to you all for free.
Was this some grand conspiracy to get everyone to play Squad? Was this an inside job? Did a developer just shut down PR for <insert some reason here>?
No, of course not.
Is PR dead?
Nope, not by a long shot and we're not giving in that easily! Stay tuned for v1.4!
I hope that covered most of everyone's questions. If you have any other questions or concerns, let us know.
Thanks for sticking with us,
- The Project Reality Team
Firstly, it's back up. You can all go play now! Yay!
But for those still reading, I want to give you all an overview of what happened over the last week, and what's happening in the future, so this doesn't happen again (or so it doesn't happen as bad again). To start with though, I want to apologise to everyone for the downtime. It sucked, and I know how much everyone was wanting to play, but couldn't.
What is this Master Server?
The master server is what controls your online player logins, and the ability to see servers in the ingame server list. This service used to be provided by GameSpy, and many games used this. However, in 2014, GameSpy was shut down, and Battlefield 2 became abandonware. To keep PR alive, we reverse engineered the original GameSpy master server and launched a replacement service. Also Forgotten Hope 2 was invited to use our master server as well, in the spirit of community relations.
What happened to it?
Last Saturday, the master server was attacked by a DDOS attack. This caused our server provider to null route the IP address running the master server, effectively shutting it down. When the master server is down, no one can play PR online. Also, since FH2 is using the master server, they were also affected by this downtime. Obviously, quite a bad situation to be in. This wasn't a new thing to us though, it's happened several times before, where we've had outages for a few hours here and there.
Why did it take a week to come back up?
Shortly after the DDOS took down the master server, I contacted the server provider to request more information, and to release the null route, like I normally do. Unfortunately, this occurred on a weekend, and the Tier 1 support I contacted could not provide any assistance. They referred me to Tier 2 support, who unfortunately only work Monday-Friday. Not the sort of thing you want to hear. I did get some good news on Sunday though, the Tier 1 support person finally realised it was an automatic null route, and suggested that it would return within 24 hours from the original attack. Unfortunately this did not occur, so it was back to waiting until Monday.
When the server did come back up, it was just attacked again, and the automatic null route was reinstated. That was something new that we haven't had before. The attackers usually gave up after a few hours. On top of this, our website was attacked and taken down as well. So this attacker was quite persistent.
To make matters worse, I personally had to leave to go on a business trip Sunday-Thursday, and was unable to do much else apart from submit support tickets from my phone to try and get the master server online again. Unfortunately, I did not have access to do the same for the website, and the person who does failed to reply to our messages. We did eventually manage to track down a retired developer who did still have access, and we got the website back up.
The master server however was continuously attacked over and over, something like 5 or 6 times if I counted correctly, and I gave up trying to get it back up until I returned home to deal with the situation properly.
So what has been done about it?
Yesterday, we have moved the master server to a different provider, one that implements much more effective DDOS mitigation techniques. We will see how everything goes over the next few weeks, and if the attacks continue and are mitigated, or if they prove ineffective. If we continue to have issues, then we still have some more things that we can do. Changing server providers was just the easiest and quickest solution. We had hoped to get it all up and running yesterday, but we ran into some slight issues with the migration, so we had to delay until today.
What about the website and that attack?
While the master server is the more critical part of the picture, the website is just as vulnerable to this type of attack. Over the next few weeks, we will be working on migrating the website to a new server which will also be able to mitigate these attacks, as well as moving all our other services that run behind the scenes. This new server is also cheaper and more powerful. Which brings me to my next point.
What's up with donations?
As I mentioned last week, donations at the moment only pay for our website, and some other behind the scenes things like our repositories. There are two servers that cost us $319 USD / month, and the donations pay for these. On top of this, I have my own server, which provides services such as our build server, test server, file hosting, project management tools, mumble, PRSPY, various utility scripts, and up until yesterday, the master server. This costs me $99 USD / month, which I pay for out of my own pocket, and have done so for the last 5 years or so. There is also a CDN which I also pay for which hosts some critical things, but that's like $2 / month, so it's not really worth mentioning.
Anyway, for those who know how much these sorts of things cost, it's clear our costs are way over the top, over $400 / month, and for the last 3 years or so we've been trying to consolidate servers and reduce our costs. Unfortunately, our web admins have been pretty inactive, and haven't been able to get anything done about it.
So instead of waiting for the web admins, we are now actively working (over the next few weeks) to just get this done already. It's been pushed back and back, and with all these attacks it's finally pushed us over the edge where we can't deal with it anymore.
Unfortunately, issues like inactivity are a common part of working on a volunteer mod development team, where you don't get paid and you work in your free time. We just have to deal with it and try to overcome any issues that may arise.
So, we are purchasing new infrastructure to handle all our use cases, consolidate all our over-priced servers, and avoid the need for developers to pay out of their own pockets.
So how much will these new servers cost?
Our new planned server infrastructure will cost us less than $200 / month, for everything, which is quite a large difference. Over the next month or two, we will also be sorting out donations to make sure it covers everything properly, unlike before (it takes time to deal with this sort of thing properly, it's not something we can just flick a switch on). Our current donation goal of $250 / month will likely stay as is, so we can cover months with reduced income.
For those wanting to donate now, please hold onto your money for the time being. We will let you know when donations are going to the things you want to pay for.
Another bonus of the new infrastructure will be multiple people having appropriate levels of access, so we don't run into a situation like last week where we can't do anything because certain people aren't responding.
Who did the attack?
Dunno, clearly someone who obviously doesn't appreciate the work we do here, provided to you all for free.
Was this some grand conspiracy to get everyone to play Squad? Was this an inside job? Did a developer just shut down PR for <insert some reason here>?
No, of course not.
Is PR dead?
Nope, not by a long shot and we're not giving in that easily! Stay tuned for v1.4!
I hope that covered most of everyone's questions. If you have any other questions or concerns, let us know.
Thanks for sticking with us,
- The Project Reality Team
