PR Master Server Update

[AncientMan]

Hi everyone. Time for an update on the current situation with the master server.

Firstly, it's back up. You can all go play now! Yay!

But for those still reading, I want to give you all an overview of what happened over the last week, and what's happening in the future, so this doesn't happen again (or so it doesn't happen as bad again). To start with though, I want to apologise to everyone for the downtime. It sucked, and I know how much everyone was wanting to play, but couldn't.

What is this Master Server?

The master server is what controls your online player logins, and the ability to see servers in the ingame server list. This service used to be provided by GameSpy, and many games used this. However, in 2014, GameSpy was shut down, and Battlefield 2 became abandonware. To keep PR alive, we reverse engineered the original GameSpy master server and launched a replacement service. Also Forgotten Hope 2 was invited to use our master server as well, in the spirit of community relations.

What happened to it?

Last Saturday, the master server was attacked by a DDOS attack. This caused our server provider to null route the IP address running the master server, effectively shutting it down. When the master server is down, no one can play PR online. Also, since FH2 is using the master server, they were also affected by this downtime. Obviously, quite a bad situation to be in. This wasn't a new thing to us though, it's happened several times before, where we've had outages for a few hours here and there.

Why did it take a week to come back up?

Shortly after the DDOS took down the master server, I contacted the server provider to request more information, and to release the null route, like I normally do. Unfortunately, this occurred on a weekend, and the Tier 1 support I contacted could not provide any assistance. They referred me to Tier 2 support, who unfortunately only work Monday-Friday. Not the sort of thing you want to hear. I did get some good news on Sunday though, the Tier 1 support person finally realised it was an automatic null route, and suggested that it would return within 24 hours from the original attack. Unfortunately this did not occur, so it was back to waiting until Monday.

When the server did come back up, it was just attacked again, and the automatic null route was reinstated. That was something new that we haven't had before. The attackers usually gave up after a few hours. On top of this, our website was attacked and taken down as well. So this attacker was quite persistent.

To make matters worse, I personally had to leave to go on a business trip Sunday-Thursday, and was unable to do much else apart from submit support tickets from my phone to try and get the master server online again. Unfortunately, I did not have access to do the same for the website, and the person who does failed to reply to our messages. We did eventually manage to track down a retired developer who did still have access, and we got the website back up.

The master server however was continuously attacked over and over, something like 5 or 6 times if I counted correctly, and I gave up trying to get it back up until I returned home to deal with the situation properly.

So what has been done about it?

Yesterday, we have moved the master server to a different provider, one that implements much more effective DDOS mitigation techniques. We will see how everything goes over the next few weeks, and if the attacks continue and are mitigated, or if they prove ineffective. If we continue to have issues, then we still have some more things that we can do. Changing server providers was just the easiest and quickest solution. We had hoped to get it all up and running yesterday, but we ran into some slight issues with the migration, so we had to delay until today.

What about the website and that attack?

While the master server is the more critical part of the picture, the website is just as vulnerable to this type of attack. Over the next few weeks, we will be working on migrating the website to a new server which will also be able to mitigate these attacks, as well as moving all our other services that run behind the scenes. This new server is also cheaper and more powerful. Which brings me to my next point.

What's up with donations?

As I mentioned last week, donations at the moment only pay for our website, and some other behind the scenes things like our repositories. There are two servers that cost us $319 USD / month, and the donations pay for these. On top of this, I have my own server, which provides services such as our build server, test server, file hosting, project management tools, mumble, PRSPY, various utility scripts, and up until yesterday, the master server. This costs me $99 USD / month, which I pay for out of my own pocket, and have done so for the last 5 years or so. There is also a CDN which I also pay for which hosts some critical things, but that's like $2 / month, so it's not really worth mentioning.

Anyway, for those who know how much these sorts of things cost, it's clear our costs are way over the top, over $400 / month, and for the last 3 years or so we've been trying to consolidate servers and reduce our costs. Unfortunately, our web admins have been pretty inactive, and haven't been able to get anything done about it.

So instead of waiting for the web admins, we are now actively working (over the next few weeks) to just get this done already. It's been pushed back and back, and with all these attacks it's finally pushed us over the edge where we can't deal with it anymore.

Unfortunately, issues like inactivity are a common part of working on a volunteer mod development team, where you don't get paid and you work in your free time. We just have to deal with it and try to overcome any issues that may arise.

So, we are purchasing new infrastructure to handle all our use cases, consolidate all our over-priced servers, and avoid the need for developers to pay out of their own pockets.

So how much will these new servers cost?

Our new planned server infrastructure will cost us less than $200 / month, for everything, which is quite a large difference. Over the next month or two, we will also be sorting out donations to make sure it covers everything properly, unlike before (it takes time to deal with this sort of thing properly, it's not something we can just flick a switch on). Our current donation goal of $250 / month will likely stay as is, so we can cover months with reduced income.

For those wanting to donate now, please hold onto your money for the time being. We will let you know when donations are going to the things you want to pay for.

Another bonus of the new infrastructure will be multiple people having appropriate levels of access, so we don't run into a situation like last week where we can't do anything because certain people aren't responding.

Who did the attack?

Dunno, clearly someone who obviously doesn't appreciate the work we do here, provided to you all for free.

Was this some grand conspiracy to get everyone to play Squad? Was this an inside job? Did a developer just shut down PR for <insert some reason here>?

No, of course not.

Is PR dead?

Nope, not by a long shot and we're not giving in that easily! Stay tuned for v1.4!

I hope that covered most of everyone's questions. If you have any other questions or concerns, let us know.

Thanks for sticking with us,
- The Project Reality Team

[AncientMan]

Donations Update:

Donations are now going to a shared PayPal account between Management ([R-DEV]AncientMan and [R-DEV]AfterDune). If you want to donate, you can now donate knowing it will be paying for our future server infrastructure.

As of the time of this update, there was $300 donated for this month. This went to the previous donation account, held by [R-DEV]2Slick4U, and is not accessible by anyone else. While that sounds somewhat bad and suspicious, it's not. This money will be used to pay for the current servers (until we switch over to the new servers) and realitymod.com domain name registration.

Thanks

[wretchedegg]

Thanks for all your hard work, man. We all appreciate what you're doing for the whole community.

[Godskalken]

Interesting read. Awesome work AncientMan, im very thankful and impressed. Donation coming on my payday.

[piratepengu]

1.4 is coming

[Bluedrake42]

I facepalmed so fucking hard when I heard the Squad conspiracy.

... also I bet Cisco was the one who did it lol

[SyriousT]

Thank you very much , waiting for 1.4

[timtherifleman]

Thanks guys. I missed my drug

[XSHADOWGX]

Thank you very much guys!!!!

[Danesh_italiano]

So... the DDOS protection of OVH works good? Or the PRMaster dont get DDOSed yet ?

Theys use Cisco Firewall Network

[BlackGus]

What kind of mental illness someone/somegroup has to have to try to shut down a small player community of less than 500 people total, and keep trying to keep it down for WELL OVER A WEEK?

Seriously, this isn't just disliking the game, this is having no life and trying to put so much effort in something so meaningless.

People from EA?? ohhh shiit Swat team in my house! help asdsadddddddddddd

[Chefmoto1]

You guys do great work, thanks for getting everything fixed and back up.

Food was starting to lose it's flavor without my PR fix.

[PLODDITHANLEY]

Thanks as ever AM it will be really nice to get the whole servers/forums/depository mess sorted out with far better service and value for money. I hope you can all work out a system that means one person going AWOL doesn't mean the end of the PR world.

I'm standing by to do my bit to help fill the PR coffers when it's sorted.

[Rhino]

I facepalmed so fucking hard when I heard the Squad conspiracy.

You didn't happen to get that on webcam by any chance?

Thanks as ever AM it will be really nice to get the whole servers/forums/depository mess sorted out with far better service and value for money. I hope you can all work out a system that means one person going AWOL doesn't mean the end of the PR world.

I'm standing by to do my bit to help fill the PR coffers when it's sorted.

Ye, if anything in the long term I believe this whole incident has in fact only made PR stronger and more robust, since the website and server issues have been a running problem in PR for some time that have been avoided for to long, but this whole thing has forced us to deal with it, and we are the better off for it!

[Evox]

Thank for all your work! Love this game!

[gwa1hir]

so at the end at least something positive grew out of it. you finally get new infrastructure going for a much better price. and hell it was about time

thanks for solving this, long live PR !

[maranov]

Great! Thanks devs!

[spain]

For those wanting to donate now, please hold onto your money for the time being. We will let you know when donations are going to the things you want to pay for.

Will do! Thanks for everything you guys do, looking forward to 1.4 and many more hours of fun.