If you are like many of our industrial clients, you'll be taking some or all…
Occasionally, when we are working on a new client’s website, whether that is redesign, minor changes to make it responsive/mobile-friendly, repairing a broken site, or moving it over to our hosting, we’ll run across a few previously-unknown issues. Actually we’ve come to expect this.
When people leave their old hosts or developers, there is usually a reason. We’ve had clients come to us with sites that were patched together to operate in substandard environments, and sites built on CMS that are no-longer supported – patched together over the years to operate in standard environments to the point where there is more patch than there is software.
We also tend to gain clients who want their site hosted someplace where their site is not one in a few hundred, or one in a few hundred-thousand sites hosted. They can’t afford for their site or email to go down, and to go a day without notice.
Sometimes we’re just liberating a client from a pointlessly expensive or otherwise problematic situation.
This is a story about in many, many site moves of this past year…. all of them interesting in their own way, but I took some extra time out to document this one…
Before the Big Move
Generally, if their site is not working – I’ll try to fix their site before migrating it over. I’ll at the very least update the software and run a security scan before downloading and exporting. In some cases, however, where the server/host is the problem, the site cannot be worked on in place.
How to be a Bad Web Host: Taking the “Control” out of “Control Panels”
A new client of ours had a website that had suddenly stopped working. When I took a look at the control panel, well… first thing I found was that I did not like the Host’s control panel – it was a proprietary mess, laid out horribly, very limited in functionality compared to most, and many of these functions did not work at all. Another thing I found was that the client’s files were still there and intact, just that PHP processing had been turned off for the entire site.
Typically, when a host just turns functionality off for a site, it is a pretty-good sign that the site has been hacked, or is otherwise misbehaving. Hosts will switch off/disable infected sites or sites that are causing issues with the server, but… one would hope that if they switched it off, they would have noted why. In this case they didn’t email the customer to tell them that they disabled this site, one the host was still billing for. I suppose making notes was also just too much work, because they apparently had no idea why it was turned off, and non-ceremoniously turned it back on.
By the time they managed this, I had already downloaded the site, and exported the database, but it was good to have the site working again while we went through the process of transferring the domain. When I ran into some problems with the domain administration in their control panel not working, I read up about the host. From there I knew was going to be a very and slow painful process… which it was. I think the former host’s only strategy for keeping customers is to make it very hard, near-impossible for customers to get away.
WITH ICANN, YOU CANN
Having mentioned my willingness to go through ICANN to make the switch happen, suddenly we had cooperation and the domain was unlocked. I still had to wait another 7 days for the former host to not contest the final transfer, because of course they were not going to use their energy to approve it – but that gave me a little time to set the site up in its future home.
By then, I had already created a new database, imported the tables, set up database users, set their permissions, pre-configured the domain pointing, and uploaded their site so that everything could be perfectly in place when the domain switched hands.
Watch Where You Put that WebSite – You Don’t Know Where it’s Been!
Since the hosted site was not working on the server when I downloaded the site and exported the database, and I hadn’t the chance to upgrade the software or run a security scan, I decided it might be good to look through some of the files before the site went live. Looking for possible backdoors is pretty important at this stage, because we definitely don’t want to bring those over to our server.
When doing this, hunting for back doors in-particular, one would think the easiest solution is to look for the most common signature: Base64_decode, but as you see below (what I found on the old site) – this is often scrambled like a sunday morning word jumble, strtolower is used to select characters from the jumbled letters in the first string into commands.
PHP Alphabet Soup
How this word jumble works is to use the help of the command ‘eval’, to make this:
With this in place, a bot or hacker, can send parameters through HTTP POST such as:
n764b3b='ZWNobyAnMW9rMScuIlxuIjtleGl0Ow==', which becomes:
base64_decode('ZWNobyAnMW9rMScuIlxuIjtleGl0Ow=='), which becomes:
Now whoever has sent this command knows their exploit is in place, because instead of the page they are ‘visiting’ is a blank page that just says “1ok1”.
This allows them, and others know that they can send pretty much any command they please through your site. This can include writing new files, using your mail server, any number of things, but any number of these typically ends up in damaging your domain’s search reputation or your domain’s email reputation. In most cases I’ve found SEO Spam (mostly Pharma Hacks), Malicious Redirects, but in some cases I have found Malware Delivery Systems, Attack Site or Referrer Spam automations, Phishing Pages, and Email forms to send Spam by. Wordfence covers that list very well here.
Searching… Seek and Destroy!
When searching for more instances of this infection, you could do a search for the whole PHP script – but you’ll likely only find the one infected page that way, the one you are already viewing. There could be hundreds of infected files in root folders, upload folders, theme/template folders and many other places.
In some cases, the original word-scramble string changes order and often name and the order in the strtolower command changes accordingly, and there could be twenty to a hundred parameter names for the the hacker to use on your site. In others, the variable names change, or there are multiple parameters that can be passed to the site (a different script for each one).
There are ways through SSH to use commands such as ‘grep’ to seek and replace this section of code out with wildcards. It can be handy in a pinch, if your host allows you this level of access, and if you formulate your command very well. Otherwise: in one shot you could accidentally remove many important lines of code from many important files across your domain; You could also end up leaving snippets of code in place that also end up breaking the site. The linked example is a how-to on fixing an infected Drupal site, but the same technique could be used for just about any CMS. Of course if you have a WordPress site that is up and running, and can install Wordfence, that is one of the quickest ways to find and remove these infected files.
One downside to working on the site in place on the server, is that backdoors could be exploited while you are fixing the site. Missing just one could put you right back in the same place again weeks, days, minutes later. If you are using Wordfence – just do a new scan after you fix the infected files and you should be fine. If you are seeking out the files and changing them by hand, you should download the site and edit files locally. You can upload the fixed website in place of the infected one when done and know that no new files were infected while you were working.
When doing this, I tend to start by searching for ‘eval’ – it’ll bring up a number of false positives, because eval is fairly-commonly used, but it will also bring up all the infected files for this type of infection. Once you’ve found all these files, then look through those files and look for commonalities in the infection other than ‘eval’.
A Common Thread… or Rather String…
In this case, I found that all of the infected files did use two common string names: $s20, and $s21. Both are present in all instances, so I only needed to look for $s21 from here, and this filters out all of the false positives.
Above: Searching an entire folder with the “Find All” command (do not do “Replace All”). This will open all files infected. You don’t need an expensive WYSIWYG, but it is nice to have this one. Any open-source text editor with a Find/Replace function should do. If you are looking for an open-source WYSIWYG, such as Brackets, that should also do.
I found around 40 files that were infected, so I just opened them all and cut this line of code out by hand. If there were more of these, say hundreds (which I have found before) – I’d have put the site into a test/quarantine server, and used SSH to search and replace.
Of course when sites are in WordPress, there are a lot of shortcuts you can take when fixing by hand, which come down mostly to where the infections reside:
- If the infected files are mostly in “uploads”, one can delete all the php files found in that folder and subfolder, and put a blank “index.php” file back into each folder. There is no reason php files should be in this place. Searching this folder on your mac or pc means just being able to highlight all the found php files and delete them.
- If they are mostly in the wordpress install itself: Delete the admin and includes folder and upload new. Upload new versions of the files in the root folder. Delete any files in the root folder that do not belong (php files that were not replaced by the new wordpress files, excluding config.php). Check config.php for malicious code.
In the above: You’ve just saved yourself from searching the root, admin, and includes folder. This should leave only the wp-content folder, for which you’ve already taken care of the uploads. The upgrades folder should be empty, so only the themes folder remains.
- Delete Themes. I tend to delete every theme I am not actively using. This means less themes to search for infections now, less themes in the future to keep updated, less themes to provide vulnerabilities to new/unknown exploits.
With those steps, you’ve saved yourself a lot of time searching through folders and files… but, if your WordPress site is hosted, and running, just install Wordfence and run the scan. You’ll save a lot of time now, and later.
Wrapping Things Up
If there were backdoors found on your site, there is a chance that the site could have been used for more than just running commands through. You’ve stopped them from getting in this way, but there can still be email forms and phishing pages, other remnants of the infection you’ll want to find and get rid of.
Don’t expect the created or last-modified dates on these files to be accurate – these can faked.
Your best bet is always being very familiar with whatever CMS you prefer to use – familiar enough to know how to wipe most files clean, replace them with new, and spot files that are out of place.
I choose to use WordPress in most cases because of my familiarity with it. I install, design, and manage a lot of WordPress sites – and have been doing this since its earliest versions.
In other cases, I often recommend managed CMS solutions where such security headaches are for the providers of the service (we build a lot of WordPress sites, but we use Hubspot for ours, and offer development and maintenance of Hubspot sites, as well as managing Inbound Marketing campaigns). There are by the month fees for these, but in many cases these can come with incredibly handy Marketing tools for the money, and save you the cost in time or money that occurs when your site is hacked.
Consider how You Might have been Hacked and Prevent It
Oftentimes, this could be as simple as having installed a plugin or a patch. Here are some tips to avoid that:
- Download plugins only from respected sites that monitor for malware plugins.
- Try to find plugins that have thousands of active users and are regularly updated.
- Don’t add a patch you’ve found on the web unless you are sure of what it does and why, or at least make sure the site you found that patch on regularly monitors for people posting malicious code as fixes.
- Tighten up security: Add a security suite if one is available for your CMS. Make sure your .htaccess is solid against browsing of folders and that it blocks forbidden files.
- Make sure your own code sanitizes strings/escapes special characters from input and from HTTP POST requests.
- Make sure your CMS and plugins are kept up to date.
- Check on your site regularly for anything out of place.
- Search Google with Site:yoursite.com to see if there are phishing or pharma pages on your site.
- Consider any new problem that did not occur with updates or changes to be a possible hack.
- Connect Google Search Console/Webmaster Tools to your site as a means to monitor for infections or other problems.