SyntaxBomb - Indie Coders

Languages & Coding => Monkey => Topic started by: MikeHart on July 25, 2020, 10:27:48

Title: Cerberus X website down
Post by: MikeHart on July 25, 2020, 10:27:48
Hi folks,


thanks to some a..hole, our website is down and I don't know how long it will take to come back, if the database is damaged and so forth.
Title: Re: Cerberus X discord server
Post by: MikeHart on July 25, 2020, 11:34:46
Of course, this place here is fine too.
Title: Re: Cerberus X website down
Post by: iWasAdam on July 25, 2020, 11:56:28
my faith in you and my thoughts - It will take as long as it takes ;)
Title: Re: Cerberus X website down
Post by: Derron on July 25, 2020, 11:57:09
Managed hosting = daily backups (most often).
Just in case you did not think of (login and check if you can "restore" something).

Also some hosting setups allow to do daily backups on your own (including mailing stuff to you) depending on what you set up in the past (and forgot about it) you might even have backups here and there.


All the best.

Ron
Title: Re: Cerberus X website down
Post by: MikeHart on July 25, 2020, 12:15:53
Ron,


guess what. My ISP does these backups automatically. Webspace AND DB. Webspace is restored, all faulty files removed, passwords changed. The DB will be restored if I notice any damages.


BUT...


for this I need my ISP to reanable the domains/Webspace. They have deactivated ALL my domains. And from what I gathered in the net, that can be a lengthy progress. IF I will succeed.

I am not even able to reroute my domains to a different place. Well I set them too, but nothing happens.
Title: Re: Cerberus X website down
Post by: Derron on July 25, 2020, 12:25:34
Hmm this stuff of "deactivation" ... I once received a letter (not an email) from Strato (must be ~10 years ago) in which they informed me that my webhost is used to send out malicious content (someone dropped a backdoor/shell script). I needed to sign that I do whatever is possible for me to have it fixed and that it wont happen anymore.

Normally it is up to you what you do with your webhost (assuming it is "legal" in the hosters country) but they might hav TOC (or AGB ;D) allowing them to suspend your account when they identify "doubtful" actions.


You could meanwhile create a "cerberusx.us.to" domain or another dynamic ip service which points to the IP of your webhost (assume you run a (v)server, not just webspace). In the hosting you needed to setup what website to "show" when using this domain (you can have virtual hosts - so depending on the domain you use another "directory" is served - this way you only have one IP but eg 10 different domains and websites).


Nonetheless: good to have a restored backup. I would urgently check your scripts for vulnerabilities. Find out how they intruded your system (I think of either your forum software or one of its addons). Especially if it was a "automatic hack" (script checked your website, found a vulnerability, exploited it and placed some automatic malicious scripts ----- compared to a manual hack which did some indidivual stuff to you like "greeting all cerberus users" or so). These "hack bots" will just return and do it again if you do not fix the security holes.

bye
Ron
Title: Re: Cerberus X website down
Post by: Qube on July 25, 2020, 14:22:16
That's really crappy for some twat to hack a site that has zero benefit to them for doing so. Probably some script kiddy who's got a script looking for that particular forum all over the web so he can boast to his "online friends" that he's hacked a site ( using someone else's tools ). Such a legend :(

Hope you get it all up and running soon again and find / fix the way they got in.

Wonder if we're next  ::)
Title: Re: Cerberus X website down
Post by: MikeHart on July 25, 2020, 15:24:54
Quote from: Derron on July 25, 2020, 12:25:34
Hmm this stuff of "deactivation" ... I once received a letter (not an email) from Strato (must be ~10 years ago) in which they informed me that my webhost is used to send out malicious content (someone dropped a backdoor/shell script). I needed to sign that I do whatever is possible for me to have it fixed and that it wont happen anymore.

Normally it is up to you what you do with your webhost (assuming it is "legal" in the hosters country) but they might hav TOC (or AGB ;D ) allowing them to suspend your account when they identify "doubtful" actions.


You could meanwhile create a "cerberusx.us.to" domain or another dynamic ip service which points to the IP of your webhost (assume you run a (v)server, not just webspace). In the hosting you needed to setup what website to "show" when using this domain (you can have virtual hosts - so depending on the domain you use another "directory" is served - this way you only have one IP but eg 10 different domains and websites).


Nonetheless: good to have a restored backup. I would urgently check your scripts for vulnerabilities. Find out how they intruded your system (I think of either your forum software or one of its addons). Especially if it was a "automatic hack" (script checked your website, found a vulnerability, exploited it and placed some automatic malicious scripts ----- compared to a manual hack which did some indidivual stuff to you like "greeting all cerberus users" or so). These "hack bots" will just return and do it again if you do not fix the security holes.

bye
Ron


1. Webhosting, so they control the domain routing. I just can wait or move everything to a different provider and move the domain.
2. They took all domains down because they had to "wegen Mitstörerhaftung verpflichtet".


How automatic it was I don't know. On my webspace I had several folders. One for CX, one for a static website, 2 for Strato hosted Wordpress sites. All domains are routed to the corresponding folders. But another unused domain was routed to the root folder. Anyway, in the root folder there was an updating.php script. The content looks like it is related to wordpress and had some big crytic strings. Then they planted some other php scripts within the cgi-bin and cgi-data folder. One looked the same like this updating.php file. The same files were inside the CX installation. The CX files and wordpress files itself were all untouched.
The none of the sites have a contact form.


According to strato they sending mass spam emails from my webspace. How I don't know. Normally they should have had access to my user login details, or? And wouldn't they need my email passwords for this?
Title: Re: Cerberus X website down
Post by: MikeHart on July 25, 2020, 15:28:33
Good news is, I was able to download the forum installation and now the DB. And it looks like no harm was done to it. But that is just judging by a look via phpmyadmin.
Title: Re: Cerberus X website down
Post by: 3DzForMe on July 25, 2020, 20:06:43
Any publicity s good publicity, opened up my Cerberus 64 today and she compiles a treat 😁👍, Kudos for keeping and evolving MonkeyX spirit. 👍
Title: Re: Cerberus X website down
Post by: Qube on July 26, 2020, 02:08:39
QuoteGood news is, I was able to download the forum installation and now the DB. And it looks like no harm was done to it. But that is just judging by a look via phpmyadmin.
That's good news from bad at least.

Also if things look like they're going to take a while then I'm happy to host on our server in your own private corner. Just give me a slap if needed :)
Title: Re: Cerberus X website down
Post by: MikeHart on July 26, 2020, 05:39:17
Quote from: 3DzForMe on July 25, 2020, 20:06:43
Any publicity s good publicity, opened up my Cerberus 64 today and she compiles a treat 😁👍, Kudos for keeping and evolving MonkeyX spirit. 👍
That is good to hear.
Title: Re: Cerberus X website down
Post by: MikeHart on July 26, 2020, 05:39:46
Quote from: Qube on July 26, 2020, 02:08:39
QuoteGood news is, I was able to download the forum installation and now the DB. And it looks like no harm was done to it. But that is just judging by a look via phpmyadmin.
That's good news from bad at least.

Also if things look like they're going to take a while then I'm happy to host on our server in your own private corner. Just give me a slap if needed :)


Thanks buddy, lets wait and see what Strato does.
Title: Re: Cerberus X website down
Post by: Amon on July 26, 2020, 08:20:36
I hope everything gets back online. Cerberus-X is a wonder to use thanks to mike. The website and forums were perfectly maintained and the community has some really cool members.

I never understood why people deface/hack websites. I guess the only logical conclusion would be the these types of people suffer from small penis problems.
Title: Re: Cerberus X website down
Post by: Derron on July 26, 2020, 08:46:56
Strato is a mess these days ... we vserver users had veeeeery slow IO sometimes. So slow that processes started hanging in dead loops and whatever, the guys of holarse linux gaming (running a multitude of game servers and the likes - on strato) had even more serious issues :D.
Strato then finally announced publically (1 or 2 months ago) that they have issues which they investigate .... wow ... you know I opened a ticket for this in last year ? and since then reinstalled my server (assumed a malicious script hiding from me), rebooted every 1-2 weeks to resolve issues for a while as even the mailserver stopped working randomly.

They "fixed" it then (after the public announce) and since then it "almost" works, I just get randomly Apache2 no longer responding (mail server works, plex as admin tool still works, ... ftp and all works, just apache is running but no longer reacting, logs do not show any error). Yeah, as the IO stuff is creeping back I assume they will "fix" it later "again".



@ Mitstörerhaftung
Yeah, this is something what I described, needed to send a letter (maybe they even accepted a "Fax").

Surely they used a vulnerability in either wordpress, wordpress plugins or forum, forum plugins. Once they have this (most often an upload or remote execution vulnerability) they place scripts and they allow to execute whatever they want (according to the restrictions of your webhost).

So similar to your forum or wordpress installation these scripts could send mails .. and mails .... and mails.


Also take this into consideration: your domains (the ones they used for spamming) will now be listed on zenhaus and other "black lists". Other webservers (like mine) contact these black lists when they receive a mail of your domain ... and if the black lists contain it, they (the mails) get rejected.

It takes a while until all the lists remove you from the active spammer list. for some you can (or even must) apply to be no longer "a spammer".

https://mxtoolbox.com/blacklists.aspx
http://www.anti-abuse.org/multi-rbl-check
....
(both do not list you ... which is GOOD!)


Writing this just in case your websites get reactivated and you wonder why eg some people cannot "register".

Also ensure: do only use forum addons which gets updates, use less but secure ones. Same for wordpress. The forum software and word press core are often well maintained (but also target of vulnerability checks ... :D) but the addons/extra modules by 3rd party are checked too while not maintained so properly in all cases.



@ why they hack
Bots ... automatic spam host take overs.


@ Qube
As good as your offer is: what happens if his websites get taken over again? Your ISP might approach to you for spamming - and even close down your line (depends on TOC of course - and your local laws). Take this into consideration.


@ MIKE
What I suggest to you is the following: move your domain to a different hosting service. First of all: domains are cheaper elsewhere than with strato, 1und1 ... etc
Second: you can just redirect your domain to a different IP / website easily.
so instead of https://h123443543.stratoserver.net (or similar) you point to a different spot on the web. Dunno if that is really feasible with "webspace" packages. I think Strato is one of the hosters now allowing "external Domains" ("Aufschaltung externer Domains"). And if they offered it I am sure they would ask some money for it.

All the cheaper hosts (https://wint.global  https://www.netcup.de) allow such stuff but might struggle with other issues. Yet having "domains" on A and "space/servers" on B allows to react on emergency cases like "B" shutting down your services for violating the TOC (eg by being hacked - or by doing dumb stuff on your own).



bye
Ron
Title: Re: Cerberus X website down
Post by: MikeHart on July 26, 2020, 09:47:49
Thanks Ron for the tips.

I will wait a few more days and then, if nothing happens, move everything to a different ISP. Wordpress I will never touch again. Imho you can't make it secure for the functionality it needs these days. Back to static websites it is.
The forum software is Xenforo, but it doesn't use the latest update as the license ran out. It is one version behind. So far no issues with that part. Might migrate it to MYBB.
Definitely I learned my lesson here. I will enable SFTP only when needed and do a write block on all parts of the webspace where no data is stored. In the forum these are the attachments of posts and avatars.
Title: Re: Cerberus X website down
Post by: Derron on July 26, 2020, 13:23:52
Of course you could remove write access to certain directories - but directory traversal attacks come from security vulnerabilities in "backends" (admin tools of stuff like a forum) or upload scripts ("user avatar upload") and the likes.

There are plenty things "to do" - eg having your scripts not running in "htdocs" etc (which is default for webspace). In most cases you can just fly "under the radar" - means you are just not popular enough to get attacked by "human beings". You only get attacked by scripts - so ensure you keep your stuff updated, hide the version of the forum software / word press (as they use it to identify - amongst blind tries - what vulnerabilities can be used, and to "look via google" who uses software version x.y.z).

If you move to a vserver/server you need to care for even more (failban to avoid ssh-logins, email server setup/config to not get rejected by others) ... securing webspace in most cases just means to patch the holes in the used software.
Static CMS is one way to go - you could even host this via github then (as a git page). For forum... yeah, use what you think suits best and is secure enough.

Then there is not much more you can do -- ok, keep your passwords non-leaked and non-trivial (login: cerberus password: cerberus).


bye
Ron
Title: Re: Cerberus X website down
Post by: MikeHart on July 27, 2020, 07:55:38
We should be up and running soon, Strato has informed me today, that they checked the webspace and activated the access to it. As right now our domain points to SB I have reset it back to my webspace and I am waiting that the change goes into effect.


Thank you for your patience.
Title: Re: Cerberus X website down
Post by: MikeHart on July 27, 2020, 08:01:41
Ok, our domain points now back to my webspace!!!  ;D
Title: Re: Cerberus X website down
Post by: Qube on July 27, 2020, 16:34:35
Great news, Mike ;D
Title: Re: Cerberus X website down
Post by: Xaron on July 30, 2020, 14:47:08
I could update Xenforo if you like? (edit: but apparently I'm not an admin anymore, which ofc makes sense if not active there ;))

Out of curiosity: Which wordpress plugin was the security hole? Because wordpress itself (if up to date) is pretty rocksolid.
Title: Re: Cerberus X website down
Post by: MikeHart on July 30, 2020, 16:35:35
Quote from: Xaron on July 30, 2020, 14:47:08
I could update Xenforo if you like? (edit: but apparently I'm not an admin anymore, which ofc makes sense if not active there ;) )
Yup, that was the reason as you didn't want to be part of the CX development anymore and concentrate on your games. Totally understandable.
Personally I think Xenforo was not the entry point. The automatic rejection of spammers signing up works great so far.
Also updating would mean new licenses to Xenforo, Resources, Media lib and the portal.
I would rather switch to a different forum.

Quote from: Xaron on July 30, 2020, 14:47:08Out of curiosity: Which wordpress plugin was the security hole? Because wordpress itself (if up to date) is pretty rocksolid.

I can't really tell you which one or  if any at all. Besides some DSGVO stuff and page builder, nothing out of the ordinary. All updated to the latest versions.

Strato couldn't really tell how the malicious scripts were planted. They also suggested that our PCs were maybe infected with a keylogger but MS Defender and 2 virus checker brought up nothing.
Maybe my wife had comments open on her blog, I don't know.
I have wiped everything besides the forum from my webspace and will part ways with Strato soon. Still deciding were to go.
Title: Re: Cerberus X website down
Post by: iWasAdam on July 31, 2020, 06:21:42
great news :) Has it allowed you to 'find the holes'? where the little beastie got in?
Title: Re: Cerberus X website down
Post by: MikeHart on July 31, 2020, 06:35:11
Like i said before, i don't know. Because of that we nuked our wordpress sites, an old static website , removed the ftp and sftp access, and put a write block on everything that shouldn't be written too. Changed all passwords.
Title: Re: Cerberus X website down
Post by: Derron on July 31, 2020, 07:45:47
I doubt passwords were the problem
I doubt static websites were the problem


Typical access via:
- google search for specific software versions
- - automatically "pen testing" for the vulnerabilities
- - breaching in automatically
- - automatically deploying the "shell script"
- - adding the list to the shell script to the bot army list
- - get automatically controlled via "remote" as all the other bot net slaves ...

Sometimes they also manipulate hosted website scripts to contain stuff to automatically "refetch" the shell script in case of being deleted.
If they do not want to use your host as bot they might even have something installed which manipulates every html, htm, php, ... file to contain iframes etc (this was happening to me some years ago ... leading to a lifetime ban of google adsense and a letter from strato requesting me to explain...)


Next to "software specific" testing they also check stuff for common "names" ... like "contact.php" or similar (just in case you removed version tags etc). Scripts can pretty good identify forms in websites and then try to attack them - by trying to exploit non-sanitized data ... so eg.

myhost.tld/contact.php?email=mymail@host.tld'); DROP TABLE importantDB;

might create issues if you do not properly sanitize data. Most modern code does handle all this stuff - but sometimes you simply forgot about an old script lurking around somewhere (in my case an forum-image-signature-generator was error prone ...)


bye
Ron
Title: Re: Cerberus X website down
Post by: MikeHart on July 31, 2020, 11:36:58
We have no contact form. The only access would be throught a post or reply. Xenforo itself does a file healthy check. All files of our versions are like the should be. If it gets attacked again, that forum gets nuked and we move to discord.
Title: Re: Cerberus X website down
Post by: Derron on July 31, 2020, 11:58:07
Contact form was just a sample. For forums it is "post.php" (or other stuff) which gets "checked" (penetration testing) if they accept invalid input to break sanitization efforts.

Forums eg allow "bbcode" (or markup or ...) and if this is not done properly, you could inject html code to attack your visitors with malicious javascript stuff (XSS Attacks).

But all in all it does not matter now as you are only on "webspace" so you do not have log files which would show the accesses to your stuff and the parameters they tried.


bye
Ron
Title: Re: Cerberus X website down
Post by: Xaron on July 31, 2020, 14:03:50
Xenforo is one of the most well designed and secured forum software out there. It's done by the good old vBulletin 3.x devs which left the company the years back to found Xenforo. And I still think that discord can be a nice addition but never replaces a forum, it's a chat with all pros and cons and actually Cerberus ones had a discord...
Title: Re: Cerberus X website down
Post by: MikeHart on July 31, 2020, 15:02:51
I agree with you Martin but if i have to constantly fight this shit for a hobby project, then i go the way that is less painfull. Many OSS projects support their user on Discord these days. Forge Engine, Raylib, FNA to name a few.