) about the recent downtime of the webserver.And it allows me to finally show the reason why I have that shiny blue [R-DEV] next to my name. It's not just because I asked nicely. Honest!
Over the past ten or so years, the server only received rudimentary attention, mostly on an "as-needed" base. We managed to drag it along like that for surprisingly long, but sooner or later progress is unavoidable. That moment is now.
The incident
We had an incident where ALL user passwords were overwritten to be the exact same. Obviously we noticed that fairly quickly, so as a precaution we shut down the websever to investigate and get ready to bolster our defenses in the future. As a secondary immediate - probably unnecessary in hindsight - precaution, we regenerated all passwords with cryptographically strong, random passwords.
Investigation revealed that nothing else happened to our database. Luckily, despite my initial fears that an attacker might have gained SSH access to our server, Fritz had already configured SSH access to be very safe. No way anyone gets into those without physically breaking into our home and hacking our computers.
The password change itself was done in a way that would not allow anyone actual access to your accounts. Passwords always have been stored in an encrypted fashion using multiple tokens, and only one such token was overwritten (with a faulty encryption, no less).
This led us to believe that this was either not done by a professional, or it was simply a random bot roaming the internet looking for easy prey.
What does that mean for you?
All of you will have to reset your password this one time. Just enter your email here:
https://www.realitymod.com/forum/login.php?do=lostpw
and follow the steps.
As always, we recommend using a strong password, and to not use the same password for multiple pages. Use a password manager if you're lazy like me! There's plenty of great ones out there (Keepass, Bitwarden, ...), some of which integrate very comfortably in all kinds of devices and browsers so you can be even more lazy and keep auto-filling your forms. Which obviously is a security risk in itself again, but it's a lot better than using simple dictionary words, especially if you use the same password here and for your banking services...
With this scary bit out of the way, let's see what else we did!
Why was the server down so long?
A bunch of core technologies we use are close to reaching their end of life. This is the part we knew about for months or even years, and thus I had already spent many hours evaluating and preparing the necessary upgrades, so I could apply them fairly quickly.
However, there's a bunch of things that you simply cannot do on a live system easily, such as kernel upgrades and other low level security updates. Also, doing that live could have opened new security and compatibility issues for the not-yet updated services. So it was convenient to do right now as well. That's why we locked most of the homepage away so we could developed and test the updates without risk of interfering with live usage.
So what did we do?
- Kernel and all base packages upgraded to latest
- Upgrade from PHP 5.3 & 5.6 to PHP 7+
- Upgraded forum software
- Upgraded all plugins and homepage to use safer (and more performant) code and DB connections
- Upgraded account security
- Updated email authentification so Google doesn't throttle emails by the forum anymore (no more hour-long delays in email delivery)
- Added TLS encryption to outgoing emails. Now we truly serve everything via TLS! Thanks, Fritz!
.If we did everything right, there should be very little that you will notice when browsing the homepage or the forum. I can't guarantee that, so I'm counting on you informing us about any remaining bugs you spot!
Summary
We had someone/something possibly use a PHP exploit to inject a SQL command. No account was compromised.
We upgraded the webserver and everything on it, making everything state-of-the-art secure.
A sweet bonus: The homepage and forum are now served about 60% faster than before, leading to an impressive ~300-400ms response time (TTFB) in the forum. Given the size of our database and the complexity of our code, this is amazing.
Just so you get an idea of the scale of things:
Our database contains just shy of one hundred thousand forum users and over 2 million(!) posts.
It not only includes a wealth of user data (emails, passwords etc.), but also a huge amount of knowledge regarding PR and BF2 in general, and not least of all sweet memes and raw history.
I am an advocate for transparency, as I believe only by being open about what happened and our response to it can you trust us in the future.
If you have feedback, questions or concerns, feel free to ask, I'll try to answer to the best of my ability.
